Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 29th April 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: P.Charpentier, J.Closier, E.Lanciotti, S.Paterson, A.C.Smith
Phone: R.Bernet, N.Brook, R.Graciani, M.Sapunov, A.Tsaregorodtsev
Apologies: M.Bargiotti, G.Castellani, Z.Mathe, R.Santinelli
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne, Joel - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne, Andrew - RAL and CNAF migration, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4, port IntegrityDB to DIRAC3
(50) - 01/04/08 - Joel - Investigate the software installation / availability failures of recent production jobs
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(56) - 29/04/08 - Roland, Stuart, Joel - Provide Production Manager guide
(00) - 29/04/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (46) - 11/03/08 - Angelo, Roberto - Send some test jobs to CNAF to test StoRM
Done - (51) - 01/04/08 - Joel - Prepare production tool 'wish-list' in advance of DIRAC3 week
Done - (52) - 01/04/08 - Stuart, Andrei - Production Manager training session
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Gianluca sent his apologies.
- RAL / PIC migrations - Action(04)
Marianne is absent. Nick mentioned that Raja and Derek could take care of the RAL migration and that Angelo could
oversee the CNAF migration. Andrew mentioned that the CNAF migration could be done with FTS and agreed to get
in contact with Angelo and Raja.
- Bookkeeping prototype - Action(07)
Elisa commented that almost all the existing data has been migrated. Zoltan has completed the mechanism for the
recreation of the views (now takes ~20mins). Elisa mentioned that indexes will be implemented in the database
schema to further improve performance. Elisa raised the issue of duplicated files in recent productions. Philippe
conveyed that the recently affected productions were also due to the corruption of the ProcessingDB. Elisa agreed to
perform the migration of these productions by ignoring the second job (with the same processed file) that will always
have a higher JobID. Consistency checks will now be performed on the old and new Bookkeeping schemas and
documentation will be provided. The suggested timescale for releasing the new system is June. In the intervening
period it was suggested to send XML reports to both Bookkeeping instances.
- lxgate34 - Action(27)
Andrew is working on the migration of the integrity checking tools from DIRAC2 -> 3.
- Software Area Corruptions - Action(50)
During last week it was understood that install_project was causing corruptions in the shared areas due to the
available versions of the tarfile module. Action reassigned to Joel.
- Production manager training session - Action(52)
Stuart gave a summary of the training course. There were 3 sessions, introduction to DIRAC, demonstrations and a
'hands-on' exercise. 12 production managers successfully submitted stripping test jobs to DIRAC3 by the end of the
course.
- Production Manager Guide - Action(56)
Roland agreed to provide a user guide for production managers in a similar fashion to DIRAC2.
There were some discussions about automating corrective actions for 'post-mortem finalization finished' jobs at
yesterday's meeting. This was discouraged due to the large number of causes of this failure. The remaining files for
several stripping jobs were suggested to be flushed to construct jobs with remaining input data.
2. Production Operations Issues
- File download failure on 64-bit python DIRAC API
It was suggested to ask Ulrik for more information in order to debug the problem. Most likely there is a problem in
the environment causing the 64bit failure.
3. DIRAC3 Tasks
- DIRAC distribution tar files
After some discussion it was agreed to proceed with Ricardo's distribution mechanism and introduce these
optimizations to the LHCbGrid tarball to reach a common solution.
- Reconstruction tests preparation
Stuart conveyed that a reconstruction production was created yesterday evening and test jobs were submitted to the
available sites.
- Transfer tests
Andrew commented that the major hold-up is with the migration to the new online systems. Nick had recently
checked with Radu and it appears the transfers can proceed. Andrew will start submitting transfers as soon as
possible.
4. AOB
None
There are minutes attached to this event.
Show them.