Production and PASTE Meeting, 25th November 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: A.Casajus, P.Charpentier, J.Closier, Z.Mathe, S.Paterson, R.Santinelli, M.Sapunov, A.C.Smith, A.Tsaregorodtsev, A.Zhelezov
Phone: R.Graciani
Apologies:
0. Ongoing Action List
(07) - 27/11/07 - Zoltan, Elisa - Ongoing new bookkeeping system development. Migration plan
(57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes
(83) - 01/09/08 - Stuart,Roberto - To perform glexec tests
(90) - 09/09/08 - Joel - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs
(93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB
(94) - 23/09/08 - Roberto, Marcos - Using SystemLogging service for dynamic services status monitoring
(95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test
(96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization
(106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install
(107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT
(109) - 14/09/08 - Matvey,Andrew - Expose DataLogging interface on the web page
(111) - 14/09/08 - Philippe,Alexei,Andrei - Productions with file merging and associated data distribution implementation
(113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties
(115) - 21/10/08 - Vladimir,Matvey - Update SAM Agent to monitor and manage submitted jobs and publish results for faulty CEs to the SAM DB
(116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC
(117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director
(118) - 25/11/08 - Roberto - Provide script to demonstrate GFAL thread safety issue and submit a Savannah bug
Completed Actions:
Done - (82) - 01/09/08 - Andrei/Adri - to provide information from Accounting and WMS to allow GridMap for LHCb VO T1 activity
Done - (118) - 19/11/08 - Joel - Request increased swap space for volhcb09 and announce when available
1. Action Updates
- Action(07) - Prototype Bookkeeping
To be discussed later in today's agenda.
- Action(57) - Automatic publishing of BDII / GOCDB information in the CS
Roberto will send a summary mail with a report on this.
- Action(82) - Provide accounting information to allow GridMap for LHCb VO T1 activity
Andrei commented that this point will be ongoing, after the meeting last week the implementation is agreed and there
will be some final tuning of the metrics associated with site behaviour.
- Action(83) - gLexec tests
Stuart mentioned that a simple script had been offered to the people certifying SCAS but no response was returned.
Roberto commented that the request for supporting the pilot role was refreshed at the sites but not all the Tier-1s are
doing this yet, it was suggested to reiterate this with a broadcast email.
- Action(90) - Production Output Data parameter
Ongoing.
- Action(93) - Improve RAW data monitoring tools and status of files in online runDB
Ongoing.
- Action(94),(95),(96) - Service Monitoring
Andrew suggested last week to use the Data Management FTS monitor as a guinea pig for the SystemLogging
service. The steps for proceeding are for Andrew to send the collection of errors to the logging system such that an
agent can be developed to perform the error analysis. Roberto spent some time this week on integrating the DMS
unit tests from Andrew in the SAM SE suite (this is not yet critical but can be after gaining experience with the
results). The LFC test from Andrew was also added as a sensor and allows to check the propagation of information
between LFC instances. There was a bug in the CondDB test where jobs were sent to incorrect sites to reflect the
Tier-1 CEs (now fixed).
- Action(106) - Ricardo - Produce twiki documentation for dirac-install
Ongoing, being prepared.
- Action(107) - Produce proposal for how to perform standardised installation using install_project/CMT
Ricardo will be coming to CERN in a couple of weeks to discuss the outstanding issues.
- Action(109) - Expose DataLogging interface on the web page
Ongoing.
- Action(115)
Matvey reported that Vladimir has been very busy with shift work and is continuing to work on this by hand, the
automation is in the pipeline however.
- Action(116)
Ricardo conveyed that the information from his CPU studies would lend extremely well to evaluate a benchmarking
factor for CPUs. Of 6k originally stalled jobs resubmitted for 3080 ~25% entered the same state, it appears there
could be some infinite looping and Ricardo will perform further studies before contacting Gloria.
- Action(117)
There was some discussion regarding how to deal with a mixture of sites accepting generic and non-generic pilots.
The suggestion was to refuse to send any LHCb user jobs to sites not supporting generic pilots (thereby adopting a
secondary mask). The other option is to use VO software tags and it was agreed to proceed in this way ensuring that
tags can be published remotely, using the DIRAC site name to guarantee the same picture for all relevant CEs listed in
the CS.
2. Production Status Report
Ricardo raised the point about GFAL and thread safety. The suspicion is that the OpenSSL in the underlying globus
tools is causing the problem but it is unclear who can fix this. Roberto agreed to provide a script to demonstrate the
thread safety issue and submit a bug. ( ACTION: Roberto )
- Report from the Production meeting
A Recent failures at some sites highlighted a problem with file locking in NFS file systems, Stuart pursued a
workaround with Marco Cl. and a new release of the LHCb package is expected in early December. Thomas reported
some consistency problems and minor typos in the Bookkeeping so these should be fixed.
Andrew summarized that the ongoing staging activity highlights CNAF and GridKa are currently below the expected
transfer rates. IN2P3 is not currently working and a direct ticket was submitted to the local admins. The staging
activity will be ramped up to the nominal rates in the coming days. Ricardo pointed out that the number of jobs for
this activity approaches the limit of the gLite WMS and Andrew agreed to reduce this number by grouping more files
together.
3. DIRAC3 Tasks
- Plans for the new BK service
There were some discussions regarding the BK migration plan, Zoltan attached a link to the agenda. Feedback has
been received from users regarding the GUI and Matvey is preparing the web GUI in parallel. It was decided to
announce the standalone browser to a wider range of users on lxplus to get further feature requests.
4. AOB
None
There are minutes attached to this event.
Show them.