Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 29th July 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, A.Casajus, P.Charpentier, Z.Mathe, S.Paterson, R.Santinelli, V.Romanovsky, M.Sapunov, A.C.Smith, A.Tsaregorodtsev
Phone: R.Bernet, N.Brook, R.Nandakumar
Apologies:
(07) - 27/11/07 - Zoltan, Elisa - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos - Develop a general service monitoring framework leading to a virtual control room
(57) - 24/06/08 - Vladimir - Automatic publishing of BDII information in the CS using GOCDB site names
(59) - 22/07/08 - Ricardo - Add ldap python modules to DIRAC python distribution
(61) - 22/07/08 - Andrei - Investigate request generation problems in the workflows
(62) - 22/07/08 - Roberto - Check instabilities in production gLite WMS instances
(63) - 22/07/08 - Roberto - Develop DIRAC3 script for adding new users to the VO
(64) - 29/07/08 - Philippe, Andrew - Prepare SRMv1 retirement migration plan
(00) - 29/07/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions:
Done - (58) - 15/07/08 - Marianne - Clean up extra 3M events for production 2146
Production operations actions will be moved to the Monday and Thursday meetings, these include:
OPS - (01) - 06/11/07 - Marianne, Joel - Remedy input data not available and catalog access issues then update DIRAC2 ProcDB
OPS - (55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
OPS - (60) - 22/07/08 - Vladimir - Develop tools to check stalled jobs for outputs and clean up as required
1. Production Issues
- Action(01) - ProcDB clean-up
Nobody was available to comment further on this action.
- Action(07) - Prototype Bookkeeping
Zoltan is waiting for Elisa to update the new productions to the new schema of the prototype bookkeeping. A GUI is
being developed for the processing. Zoltan commented that the interfaces are synchronous for the insertion in the
database.
- Action(09) - Virtual Control Room
Roberto is converting a sensor from Raja to report the status of the VO-boxes. Discussions are ongoing with Marcos to
establish sensors based on logging information. Philippe raised that the SE testing reports to SAM are declaring many
Tier-1s as unavailable erroneously. Roberto agreed to compile a twiki page to explain the issues resulting in test
failures, e.g. GridKa firewall issue.
- Action(55) - BK Registration Failures
There are ~4.5K remaining errors in the old BK. Currently the BK manager must be restarted weekly and this is
being investigated. The 4.5K remaining seemingly have no ancestor files available, there was a suggestion to scrap
this data from the BK, LFC and storage.
Some discussion regarding the problematic Castor tape.
- Action(57) - Automatic publishing of BDII information in the CS
- Action(58) - Clean up extra 3M events for production 2146
We agreed to keep these extra events for the physicists so no cleaning required.
- Actions (57), (59) - Add ldap module to distribution
Ricardo should comment later today on the status of this.
- Action(62) - Evaluate gLite WMS instances
Temporarily using one additional WMS instance from ATLAS for testing. Adri reported that another instance at CNAF
has now become usable. The LCG IS should be cleaned to convey the gLite WMS instances that we have and are
usable. Roberto conveyed that the possibility exists to convert our LCG RB instances to gLite WMS but we agreed to
wait until the DIRAC2 retirement for solving this.
- Action(63) - DIRAC3 scripts for adding new users to the VO + CS
Roberto mentioned a prototype of this script was made available in bash but the DIRAC python script is pending.
Stuart reported that productions 37 and 38 were extended in DIRAC3 for the following requests from the CP WG and
these are running fairly smoothly. Vladimir reported an issue with some jobs stalling at around 4am and this is
pending investigation. One important request was for separating Production and Development gLite WMS's to avoid
destructive interference, this was already performed by Adri. Due to the 'has replica' flag not being conveyed to the
Bookkeeping for productions 37 & 38 this will be updated by hand. The BK catalog plugin should be carefully checked
to ensure that this corrective action is not required in the future.
2. DIRAC3 Tasks
- DIRAC release
The development release now includes the new proxy framework (embedding DIRAC groups inside the proxy). There
is now a proxy management system with a service to update the proxies. OpenSSL is no longer required on the
command line, pyGSI is now used. Delegation is used everywhere and we are now in full compliance for the pilot WG
document. Combined software installation will also be deployed with the release and this will enable SAM test jobs to
run in the production system. The Bookkeeping replica registration will also be added to the release but this is
pending the DIRAC installation on volhcb07. After the release it was agreed to update the checklist.
- Service volhcbXX hosts usage
There are 4 new volhcbXX hosts, 08 - 11. Two of these will be dedicated to the DIRAC3 Production system (08, 09).
It was agreed to benefit from this experience to define the procedure for migrating the production system. The
machines to be retired are lxgate34 (Done), lxgate03 (pending production release for SAM jobs), volhcb07. volhcb03
should also be retired so the DIRAC3 development system can be moved to volhcb01 and 02. The prototype
bookkeeping will move to volhcb10 volhcb11 will be used for creating a hot-swap backup of the WMS.
- WMS testing policy
There was some discussion on the testing policy. In the future any testing should be announced and scheduled for
periods when no ongoing production requests are pending.
3. AOB
Raja raised the question about the retirement about the DIRAC2 user system. Retirement of SRMv1 is pending the
retirement of DIRAC2 so this becomes important. Stuart mentioned that the delay to the development release has
put any Ganga developments on hold but this should get back on track in the coming days. A strategy for the SRMv1
retirement will be discussed offline and proposed in the coming weeks. ( ACTION: Philippe, Andrew )
There are minutes attached to this event.
Show them.