Production and PASTE Meeting, 26th June 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, A.Casajus, G.Castellani, P.Charpentier, J.Closier, R.Graciani, G.Kuznetszov, S.Paterson, R.Santinelli, A.Smith, A.Tsaregorodtsev
Phone: R.Bernet, N.Brook, A.Carbone, U.Marconi, M.A.S.Miguelez, R.Nandakumar
Apologies: D.Perego
1. Production Status Report
Minutes from the production meeting are available
here.
Joel reported the production status in Davide's absence. ~2K jobs have been running per day but not more. The
mask has been checked for consistency and sites consuming many pilots have been banned. It was agreed to ensure
every mask change is logged via an email to the production list. ( ACTION: Joel )
SAM reports ~47 sites eligible for LHCb jobs (out of ~108 total sites). Recent investigations have shown
install_project.py is creating links in the shared area. The check for installing software should remain in the Job Agent
but the software distribution module should be revisited. ( ACTION: Joel, Stuart, Hubert )
For the reconstruction, some data for several jobs have failed to be staged after 6 attempts in 24hrs. Joel has
developed a python script to extract the necessary information from the JobDB. At CERN the problem is yet to be
understood. The specified files do not exist for staging failures at GridKa and PIC. Raja agreed to look at the situation
at RAL. ( ACTION: Raja )
For the stripping, Joel mentioned that all files were re-entered into the processing DB after some cleaning by Gennady.
There have been many failures in DaVinci so far. Stripping jobs for CNAF have not yet started and this should be
investigated because e.g. software tags could be incorrect. ( ACTION: Joel )
Stripping jobs have been submitted to CERN, PIC, RAL, CNAF, IN2P3 but not to NIKHEF (SARA problem) or GridKa (in
downtime). DIRAC v2r15 is now in production so castor protocol is in use at CERN.
2. Production Operations Issues
There was some discussion on the issue of software installation and it was concluded that software should continue to
be installed in jobs. For DIRAC3 we can log whenever a job has to install the software and report any abnormal site
behaviour. Software tags can be presented by pilot agents but this will be ignored in the case where no software tag
requirements are in the job JDL.
The maximum concurrent jobs problem at GridKa is currently under investigation. Joel reported that Johan and
Michael were following these issues at a high level. The GridKa tape and SRM configurations are also being checked.
On some of the SLC4 sites edg-brokerinfo has ceased to function, a fix for this should be introduced to the pilot
wrapper script. ( ACTION: Gianluca )
Several use-cases have appeared for the shipped middleware with DIRAC e.g. NorduGrid, 64 bit platforms, LCG. It
was agreed to build a combined distribution to accommodate all use-cases. ( ACTION: Roberto, Roland, Matvey )
Philippe attached a storage
migration plan to the agenda. This stimulated some discussion and it was agreed to update
the twiki page for production procedures. ( ACTION: Philippe ) Marianne was reminded to create a twiki page for data
management operations e.g. file removal status. ( ACTION: Marianne )
Andrei reported that the proxy renewal fixes appeared to be stable. Gianluca was asked to continue the code cleanup
and affected components will be part of the next patch release of DIRAC. Print statements should be removed from
the LCG utilities. ( ACTION: Gianluca )
Stuart conveyed no response had been made regarding the v2r15 release. At the time of writing the OK has been
given by Andrew Maier. Andrei suggested we should deploy the DIRAC API on lxplus. ( ACTION: Joel )
Marianne reported that some files have been cleaned and the lists are being prepared for the remainder. Transfers to
PIC have been failing and other channels have been blocked, investigations are ongoing. ( ACTION: Marianne )
3. DIRAC3 Tasks
As mentioned last week, Ricardo compiled the following
PIC support document.
4. AOB
None
There are minutes attached to this event.
Show them.