Production and PASTE Meeting, 3rd June 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: A.Casajus, G.Castellani, P.Charpentier, J.Closier, Z.Mathe, S.Paterson, R.Santinelli, M.Sapunov, V.Romanovsky, A.C.Smith, A.Tsaregorodtsev
Phone: R.Bernet, N.Brook, R.Graciani, U.Marconi
Apologies:
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne, Joel -Remedy input data not available and catalog access issues then update ProcDB
(04) - 13/11/07 - Marianne, Andrew - RAL, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4, port IntegrityDB to DIRAC3
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(56) - 29/04/08 - Roland, Stuart, Joel - Provide Production Manager guide
(00) - 03/06/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (58) - 13/05/08 - Roberto, Joel - Resolve issue with shared area at LCG.Brunel.uk
Done - (59) - 13/05/08 - Andrei, Angelo - Check StoRM SE definition in DIRAC2
Done - (60) - 13/05/08 - Andrei - Add LFC synchronization to addDirectory call
1. CCRC Status
Nick agreed to take minutes for this session and post them to the LHCb CCRC'08 ELOG.
2. Production Status Report
Minutes from the production meeting are available
here.
- ProcDB update - Action(01)
The agent is running but work proceeds in order to remedy 'Input Data Not Available' and 'Catalog Access' failures.
- RAL Migration - Action(04)
Andrew was very happy to announce there were no tapes remaining to be migrated. There are several files leftover
from the first pass to correct and the LFC / CS will be updated accordingly. The timescale to completion is Thursday.
- Bookkeeping prototype - Action(07)
Zoltan is continuing with Oracle / AMGA tests and expects to have something prepared for the software week in two
weeks time. Philippe summarized the outcome of the recent Bookkeeping meeting, Elisa is continuing with the
migration of existing data and is implementing the suggestions from the Online group. The migration is complicated
by the ongoing DC06 activity. There was some discussion regarding the migration to the new Bookkeeping but it was
agreed to postpone any migration plans until the software week.
- Monitoring framework - Action(09)
Nothing new to report.
- Backup DMS services on lxgate34, port IntegrityDB to DIRAC3 - Action(27)
Ongoing. Philippe mentioned that integrity agents must be in place in DIRAC3 to allow the recreation of jobs after file
replication / removal. Procedures must be put in place to cope with this. For the IntegrityDB, Gianluca mentioned
~21k files are now remaining.
- Recover data that failed to be registered in the Bookkeeping - Action(55)
Marianne is absent. Andrew agreed to investigate the existing BK xml requests to see if another attempt to add these
files will recover any of the discrepancy.
- Production Manager Guide - Action(56)
Roland is on shift next week so will work on this then.
- Add LFC synchronization to addDirectory call - Action(60)
Philippe confirmed that the LFC synchronization is working but mentioned that the restart of transformations was not
enough to ensure a new mask was taken. As an aside, the file mask in DIRAC2 is limited to 256 characters, Andrei
agreed to check the status of this in DIRAC3.
3. Production Operations Issues
- DIRAC2 Production issues
Philippe remarked that it is impossible to run at IN2P3, the list-match reports 0 available CEs and this will be
investigated. The failover of log files can create inconsistent behaviour since in this case a tarfile is present instead of
an unpacked directory. This type of jobs should not be rescheduled. Shifters for DIRAC2 should not submit any jobs
until the recent patches are tested and a new release is made. There is pressure from the sites to retire SRMv1.
- Aborted pilots in DIRAC2
Ricardo reported that all pilots are being aborted with 'maradonna' errors for the first few hundred jobs in the queue.
This issue will be investigated with some urgency by the developers.
4. DIRAC3 Tasks
A summary of the issues to be resolved during the DIRAC3 week are:
- Next version of the WF modules (failover,accounting, ProcDB reporting, etc)
- Independent WF library distribution
- MC, user analysis workflows, jobs
- Setting up test system on volhcb01
- Components profiling, optimization
- LFC Replica status treatment
Ricardo raised the issue of python2.4 / python2.5 and how we can obtain python bindings for python2.5 in the
application area, Philippe agreed to check up on this point.
5. AOB
- Next Production Shift manager tutorial
Andrei and Stuart suggested that the production shift manager tutorial proposed for the software week should be
postponed such that efforts can be redirected to DIRAC3 commissioning.
Philippe mentioned that the IN2P3 VO-box is still not working after the recent update from Raja. Andrei agreed to
quickly investigate this problem.
There are minutes attached to this event.
Show them.