Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 25th March 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: A.Tsaregorodtsev
CERN: G.Castellani, J.Closier, R.Santinelli, Ph.Charpentier
Phone: R.Bernet, M.Sapunov, M.Seco, A.Tsaregorodtsev, R.Graciani
Apologies: S.Paterson
0. Ongoing Action List
(01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(36) - 12/02/08 - Roberto, Gianluca - Investigate other experiments usage and functionality of the Dashboard
(38) - 26/02/08 - Zoltan, Elisa, Marianne - Review the view-building process in the Bookkeeping
(41) - 26/02/08 - Marianne - Trigger replication of stripped data to other Tier-1s (RAL, IN2P3)
(42) - 04/03/08 - Marianne, Gianluca, Joel - Perform an integrity check on the ProcDB and submit pending jobs
(44) - 04/03/08 - Roberto - Contact NIKHEF about increasing the length of available queues
(46) - 11/03/08 - Angelo, Roberto - Send some test jobs to CNAF to test StoRM
(47) - 18/03/08 - Andrei - Update the proxy renewal procedure on the analysis system (DIRAC v2r19)
(48) - 18/03/08 - Gianluca, Stuart - Investigate the reason for multiple pilots aborted after
successful "list-match" check
(49) - 18/03/08 - Roberto - Develop and deploy a sensor checking the SAM DB site info against
lcg-infosites
(00) - 11/03/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (25) - 08/01/08 - Zoltan - Resolve why some files are double-counted in Bookkeeping
Merged - 01 & 42 - Marianne, Gianluca, Joel - Perform an integrity check on the ProcDB and
submit pending jobs
1. Production Status Report
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Gianluca had reported that ~20K files were checked and 12K more files at PIC are being
checked now. The file status is updated in the ProcDB to "unused" if still can be used.
Related to that, a point was raised by Philippe about onen third of various event types
not stripped yet. Since there are no more stripping jobs running, it was suggested to
inspect the corresponding stripping transformations and retag all the files in non "Processed"
state to "unused state". If the number of the files to be reprocessed matches the number
of events to be processed ( from Bookkeeping database ), then the Transformations should
be activated and jobs submitted. The operation should be supported by the appropriate
tools to analyse the ProcDB contents. Action: Gianluca, Joel - added to Action(42)
- Monitoring framework - Action(09)
Roberto will update the document listing all the service ( DIRAC and third party )
to be monitored together with the parameters and metrics to be watched.
- Resolve why some files are double-counted in Bookkeeping - Action(25).
The reason of the problem is understood by Zoltan. Since nothing can be done, the users
are warned that in 10% of queries the returned result on the number of files is wrong.
Action closed.
- Investigate other experiments usage of the Dashboard - Action(36)
The action is kept to investigate both Dasboard and MonAlisa usage by the
experiments in the actual production procedures.
- Queue length at NIKHEF - Action(44)
The queue was extended to 41 hours. Some stalled jobs were rescheduled. Roberto and Stuart
will check the status of those before the action point can be closed.
- Update the proxy renewal procedure on the analysis system (DIRAC v2r19) - Action(47)
The proxy renewal procedure fixed fixed in the production system should also be fixed on the
analysis system, subject to upgrade to DIRAC v2r19.
Gianluca reported many waiting jobs in the analysis system, while a lot of pilot agents
are aborted. The pilots are usually aborted due to RN failing to find a matching resource.
It means that the logic of a "list-match" call before submitting pilots is failing. This
should be investigated - Action(48)
Gianluca reported that the last bunch of SAM jobs were submitted by mistake to the
production system. Andrei requested investigation of what is to be done to move the SAM
job submission to DIRAC3 (Gianluca, Joel, Stuart )
Gianluca reported the the DIRAC2 production site mask was cleaned up from the obsoleted
site names. It was remarked that the site information available from the Sam DB and from the GOC
DB can differ. It is necessary to make a new "sensor" to check the SAM DB info against the
information available through lcg-infosites command - Action(49) for Roberto
2. Production Operations Issues
The DIRAC3 distribution tar files for SLC4-32bits built on volhcb04 cause /tmp directory
overflow. There were a discussion in the release building procedure. Some of the conclusions
are the following:
- The binaries from the AA and LHCb area should be taken where possible. This needs checking
with Stefan and Hubert if some missing components necessary for DIRAC are missing ( python
modules, libraries ). The best will be to add those components to the AA and LHCb areas
- The "standard" SLC4 binaries should be checked for compatibility with some more recent
platforms (Ubuntu, Debian, Suse) and marked as compatible where possible.
- Joel and Ricardo will define the release building procedure in details including the
machine to build releases for both 32 and 64 bit platforms.
- The DIRAC3 external packages installation should be moved to the standard SAM job
installation ( Joel ).
3. DIRAC3 tasks
The main DIRAC3 tasks to be carried out before the CCRC phase 2 start were presented
during the last software week, the corresponding presentations are attached to the
PASTE meeting agenda. Some of them were briefly discussed.
- Joel reported that the stripping workflow is starting to work but needs more testing.
Once tested and interstep connections are sorted out, the MC workflow will be easy
to create. So, full chain tests are to be started now with these new workflows.
- The Time Left and benchmarking utilities are ready for the tests and the pilot agents
are instrumented. Roberto and Stuart will prepare a note which will decribe in details
the procedure to eveluate in a batch system independent way the amount of CPU available
for jobs. The note will eventually be complemented by the practical results.
- Monitoring services and activities need more feedback from the potential users -
production and operation managers. Entry point web pages are ready, navigation from this
pages and definition of more detailed pages is under way.
- The Accounting service is ready, the job wiorkflows should be now instrumented with the
clients to pass the information to the services. Ricardo will provide the necessary description
of the accounting information structure to be used by the clients to pass it to the service.
- The Ganga developers will be contacted to start the Ganga interface to DIRAC3 as soon
as possible.
Philippe raised the point of the possibility to submit pilots directly to the CE. This should
be the main mode of operation for the DIRAC site where the gatekeeper agent is sending pilots
to the local batch system. The same can be applied with the CERN batch system. The question
of passing the pilot credentials should be sorted out. The future CE ( CREAM ) should be suitable
for the direct pilot submission o the grid.
4. AOB
The next DIRAC3 developers week at CERN will take place the week of 7th April.
During the week the functionality needed for the CCRC phase 2 will be finalized.
There are minutes attached to this event.
Show them.