Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 8th April 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, N.Brook, G.Castellani, P.Charpentier, J.Closier, A.Khelezov, G.Kuznetszov, Z.Mathe, M.A.S.Miguelez, S.Paterson, R.Santinelli, M.Sapunov, A.C.Smith, A.Tsaregorodtsev
Phone: R.Bernet
Apologies: R.Graciani
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne, Joel - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(46) - 11/03/08 - Angelo, Roberto - Send some test jobs to CNAF to test StoRM
(50) - 01/04/08 - Gianluca - Investigate the software installation / availability failures of recent production jobs
(51) - 01/04/08 - Joel - Prepare production tool 'wish-list' in advance of DIRAC3 week
(52) - 01/04/08 - Stuart, Andrei - Plan Production Manager training session
(54) - 08/04/08 - Andrew, Marianne - Investigate ~20-30K problematic files listed in IntegrityDB
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(00) - 08/04/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (48) - 18/03/08 - Roberto, Gianluca - Investigate the reason for multiple pilots aborted after successful "list-match" check
Done - (53) - 01/04/08 - Stuart - Introduce splitting of large bulk LFC queries in DIRAC API
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Gianluca reported that ~20-30K files in the IntegrityDB receive no response from lcg-gt, it was agreed to investigate
the cause of this discrepancy. ( ACTION: Andrew, Marianne ) The ProcDB updates are ongoing and should eventually
resolve the jobs failing due to inconsistency with the LFC.
- RAL / PIC migrations - Action(04)
Marianne mentioned that the PIC migration is now completed but the LFC is still to be updated. For the RAL migration
there were problems with the FTS job submissions and investigations are ongoing.
- Bookkeeping prototype - Action(07)
Zoltan is now testing DIRAC3 services against the new schema provided by Elisa. Philippe conveyed that the
migration of existing data was becoming very tricky and occurring at a slow pace. Andrei suggested that the testing
of the new schema with automatically generated data should occur as soon as possible to ensure that the new
schema is scalable.
- Monitoring framework - Action(09)
Andrei suggested that a meeting will occur during the developers week to discuss the document from Roberto. SAM
sensors for standard LCG services are being prepared.
- Test jobs to CNAF to test StoRM - Action(46)
Pending the end of the CNAF downtime.
- Investigate Aborted Pilots - Action(48)
It was agreed to close this action point after preliminary investigations yielded no results.
- Develop sensor checking SAM DB against lcg-infosites - Action(49)
Roberto explained that there was a bug in the SAM framework introducing an inconsistency in the available LHCb
sites. It was agreed to produce a test to display the information. Merged with Action(09).
- Software Area Corruptions - Action(50)
To resolve the software corruption problems it was agreed to remove everything in the shared areas and reinstall
from scratch. Gianluca will now proceed to do this. Joel requested that the latest version of Gauss was used for the
application testing jobs.
- Bulk LFC Queries in Ganga - Action(53)
Stuart conveyed that after discussions with the Ganga developers a fix has been deployed and a patch release will be
made available on lxplus, Action is closed.
Joel mentioned that there were problems reading data at NIKHEF. The latest version of dCache not being used was
the suspected cause but Joel discovered that this caused problems with the dCache libraries not being available in the
LD_LIBRARY_PATH. Joel will restore the previous GaudiApplication functionality for now and reschedule a couple of
jobs to allow the NIKHEF site admins to investigate.
Philippe raised the problem of some files not being registered in the Bookkeeping, Marianne is collecting such files and
this is ongoing. ( ACTION: Marianne )
Gianluca found that some problematic replicas had no further replicas in the ProcDB, these should be cleaned to allow
affected jobs to run.
2. Production Operations Issues
- SRM1 Retirement Planning
The proposal from the sites is a timescale of June '08 to retire SRMv1 endpoints. Andrei mentioned that backporting
DIRAC3 machinery to DIRAC2 should be avoided if possible. The retirement of DIRAC2 should be aimed for by June
but the official deadline of August was agreed.
3. DIRAC3 Tasks
Nick asked for a brief update on the Stripping related activities in DIRAC3. Joel reported that the workflow is being
updated to handle parameter passing between modules. Gennady mentioned that complex objects can now be
passed between objects and this is in the testing phase. The DaVinci and Brunel versions for the stripping were
requested to be available on the 28th April. Stuart mentioned that the StagerSystem was now functional in DIRAC3
and the basic stripping test jobs have been submitted. Andrei has reworked the Failover request management
system and this requires an update of some workflow modules to introduce the failover operations. The issue of
keeping track of problematic replicas was raised by Joel and the sequence of events should be carefully checked. The
Accounting client is not yet available but this will be clarified as soon as Adri / Ricardo become available. Ricardo has
produced an updated DIRAC installation script that should be tested. Matvey is continuously working on the webpage
and updates are ongoing. Joel has started discussions with Hubert about shipping a DIRAC version of SRM2 and
other externals. Philippe wanted to ensure that an alternative to regular expressions is provided for defining new
runs and productions, Andrei agreed to have something in place before phase2.
4. AOB
Roberto announced that the NIKHEF/SARA LFC instance has recently become available and is being tested now.
There are minutes attached to this event.
Show them.