Production and PASTE Meeting, 6th March 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, A.Smith, S.Paterson, N.Pukhaeva
Phone: M.Adinolfi, N.Brook, C.Cioffi, R.Graciani, R.Nandakumar, A.Tsaregorodtsev
1. Production Status Report ( J.Closier )
Minutes from the Monday meeting are available
here. Joel reported that simulation jobs were running smoothly,
one problematic site (UCL) was recently banned. The issues at UCL should be investigated. ( ACTION: Raja )
Through persistence, the reconstruction jobs are starting to run. Many jobs are reaching the maximum number of
reschedules due to the long wait for file staging. Joel reported that so far Lyon and RAL are the only sites which have
not run jobs. The affected reconstruction jobs should have the reschedule counter manually reset to zero. ( ACTION:
Andrei )
2. Production Operations Issues
Andrei reported that DIRAC v2r13p5 has been released. Release notes are available in the DIRAC CVS repository.
Last Friday there was a meeting to discuss the problems with DCache sites. It was found that lcg-gt fails to trigger
the staging of files (GGUS tickets have been submitted). In the mean time a workaround for the job wrapper has
been implemented by Stuart, making use of the 'dccp' functionality reported by Andrew. In the case of gsidcap TURLs
being returned after the lcg-getturls call, this will execute a multi-threaded 'dccp' for the files and check the metadata
to ensure staging has been successful. This has been committed but requires testing. ( ACTION: Stuart )
Ricardo reported that the VO-boxes are starting to clear the backlog of transfer requests. Two SEs are full but running
jobs can still transfer their output there so the logic should be updated to prevent over-filling the SEs. A list of banned
SEs should be passed to the Replica Manager and this should be entered in the central configuration service. (
ACTION: Andrei ) The mask monitoring page should be updated to display the banned SEs ( ACTION: Raja ). It was
also agreed to develop a site banning mechanism based on job types.
Another issue with the VO-boxes are where some RDSTs end up on disk at the correct site instead of on tape. It was
agreed to just leave the file on disk but Andrei mentioned this needs a bit of development in the Transfer Agent since
the transfer requests will have to be ignored. ( ACTION: Andrei ) The thread limits were discussed and a limit of 100
Transfer Agent threads per VO-box and 20 parallel transfers per site were decided upon. These settings should be
updated in the central CS. ( ACTION: Andrei )
Gianluca reported that after putting the gLite Agent Director into production, 13K jobs were submitted to two gLite
RBs which caused problems. It was agreed to scrap these jobs and drain the RBs since jobs were failing. A bug in the
gLite wrapper script meant that jobs were run in the user area instead of the shared area and subsequently ran out of
space. It was agreed to suspend submission through the gLite Agent Director until this bug is fixed.
The Bookkeeping services have all been deployed to volhcb01. Carmine mentioned that the faulty view problem was
resolved and now the views are updated every evening. The new Bookkeeeping page has been deployed for testing.
Marianne has updated the SE names for DC06 data in the LFC and reported that this process was ongoing for the
DC04 data. Andrew confirmed that the list of SIM files to scrap has been distributed but the catalog removals were
still to be performed. Andrei mentioned that all SIM data could be scrapped from Lyon HPSS. The remaining files can
be removed after checking that recent transfers are from old requests. ( ACTION: Andrei )
The CNAF SRM endpoint is set to change and so the LFC developers should be contacted for changing the SURLs. (
ACTION: Marianne )
3. DIRAC3 Tasks
Ricardo stated that the changes discussed during the SW week to the frameworks were being made. The timescale
for this is a few days. ( ACTION: Ricardo, Adria)
Joel reported he build tools for DIRAC3 are under development. ( ACTION: Joel )
Andrei asked Stuart to provide a page describing the use of Epydoc. ( ACTION: Stuart )
4. AOB
None
There are minutes attached to this event.
Show them.