Production and PASTE Meeting, 10th April 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, A.Smith, S.Paterson, N.Pukhaeva
Phone: R.Bernet, R.Graciani, R.Nandakumar, A.Tsaregorodtsev
Apologies: P.Charpentier, J.Closier
1. Production Status Report
There was no Monday meeting due to the Easter break. Stuart reported the progress with Stripping and
Reconstruction jobs submitted over the weekend (Production IDs 1649 and 1849-52):
Reconstruction
- 203 jobs sent to NIKHEF: 78% success (remainder maximum rescheduling reached).
Stripping
- RAL: 2 jobs, 1 success 1 failure
- PIC: 1 failed and 1 waiting job (remainder maximum rescheduling reached)
- CNAF: 109 jobs sent, 89% success (2 stalled jobs and some post-mortem finalizations)
- NIKHEF: 25 jobs sent, 92% success (remainder maximum rescheduling reached)
- GRIDKA: 14 jobs sent, 50% success (jobs stalled or post-mortem finalization finished).
There was a staging problem at GRIDKA ~150 files yet to be staged due to problematic tape but this is now resolved
and the remaining ~60 stripping jobs are starting to execute. Overall, there were > 22K files successfully staged at
the Tier-1 sites over the Easter break.
The failures at NIKHEF correspond to the Globus library problem, Andrei suggested this should be added to the DIRAC
distribution. ( ACTION: Andrei ) Stuart agreed to investigate the failed jobs. ( ACTION: Stuart )
Raja confirmed that all pending Stripping and Production jobs had been submitted and it was agreed to wait for Joel to
return before constructing new jobs.
2. Production Operations Issues
Stuart confirmed that the Stager was stable and can now handle high loads. Some changes were made over the
weekend to work around srm-get-metadata failing for large numbers of files. Andrei asked about the status of data
access at PIC, Stuart agreed to follow this up. ( ACTION: Stuart )
Regarding the data accumulation at CERN, Andrew mentioned that 20K files had been transferred to CERN-disk over
the weekend. Marianne agreed to prepare the list of files for FTS jobs by the end of the day ( ACTION: Marianne ).
Andrew reported that RAL has a large portion of the files and transfers have so far been slow from there to CERN.
SARA is also displaying poor response times. Since this is on the critical path before the cleaning of Tier-1 disks
Andrew was tasked to submit GGUS tickets for both RAL and SARA ( ACTION: Andrew ). An update on the status will
be given at the Thursday meeting. Ricardo requested that the URL for the transfer accounting be fixed, Andrew has
since confirmed that the new URL is functioning.
Due to the Easter break, Andrew has had limited time to spend on the zero size file investigation but agreed to report
on this at the next PASTE meeting ( ACTION: Andrew ).
Gianluca conveyed that there is now a consistent treatment of VOMS proxies in the WMS except for places where
grid-proxy-info is explicitly called. Andrei suggested to clean this up over all the code and add a voms-proxy-info
method to the LCG.py ( ACTION: Gianluca ) Once this is done, the User system can be migrated to volhcb01 and we
can announce to the experiment the switch to VOMS proxies.
Roland reported that no files were visible in the Bookkeeping for recent productions. Andrei suggested that this was a
problem with the views which has since been resolved by Carmine.
3. DIRAC3 Tasks
In order to formulate a job prioritization approach Gianluca was asked to collect different strategies (CMS, ATLAS etc.)
and make a proposal. ( ACTION: Gianluca ) The possibility of using Maui should be pursued as one option and Gianluca
confirmed that contact had been made with Maui support.
Andrei attached a
proposal for the DIRAC3 Configuration Service schema which stimulated some
discussion. The changes to the first half of the document are to be made before discussing the second half next week.
( ACTION: Andrei )
The Framework and Building Tools status is unchanged from last week. Andrei has updated the WMS instances to the
new MySQL class and has started the DB migration.
lxgate14 can now be retired after the shut down of the Configuration Service. The issue of the CS slaves on the
VO-boxes is still to be resolved. ( ACTION: Ricardo, Raja )
4. AOB
None
There are minutes attached to this event.
Show them.