Production and PASTE Meeting, 10th March 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: J.Closier, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith, F.Stagni, A.Tsaregorodtsev, R.Vazquez
Phone: R.Graciani, A.Zhelezov
Apologies: P.Charpentier, Z.Mathe, M.Sapunov
0. Ongoing Action List
(57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes
(83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification)
(90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs
(94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test
(96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings
(109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (pending lhcbweb deployment)
(111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side e.g. LHCbJob
(126) - 20/01/09 - Andrei - Document the design of the workflow / production management components
(132) - 03/03/09 - Stuart - Update LFN construction utility to store production output directories as a parameter
(133) - 03/03/09 - Matvey - Provide request DB interface page
(134) - 03/03/09 - Matvey,Andrew - Provide RunDB monitoring page
(135) - 03/03/09 - Andrew - Send accounting from Online DIRAC instance via the gateway to PIC
(136) - 03/03/09 - Vladimir,Roberto,Joel - Integration of existing SAM infrastructure tests into DIRAC SAM framework
(137) - 10/03/09 - Ricardo - Implementing banning specific job types
(138) - 10/03/09 - Andrei - Make a proposal for coping with problematic Tier1 sites absent for long periods during data taking
1. Action Updates
- Action(57) - Automatic publishing of BDII / GOCDB information in the CS
Vladimir reported that CS information is filled via a script, this will be put into an agent as soon as possible.
- Action(83) - gLexec tests
Testing is ongoing. There seems to be authorization failures at NIKHEF and GridKa.
- Action(90) - Production Output Data parameter
The AnalyseLogFile / BookkeepingReport should be updated, Stuart will proceed with this in the coming days.
- Action(93) - Improve RAW data monitoring tools and status of files in online RunDB
Split into (133,4,5).
- Action(94) - SystemLoggingDB example
Ongoing.
- Action(95),(96) - Service Monitoring
Roberto is working on the unit test from Andrew to run against all space tokens.
- Action(109) - Expose DataLogging interface on the web page
Ongoing. This is not yet deployed in production.
- Action(111) - File merging productions
No progress here, the recent efforts have been on the production request GUI.
- Action(116) - CPU studies
Ongoing.
- Action(122) - Develop specialized SAM Agent to track and report errors with regard to pilots
Ongoing.
- Action(124) - Apply dataset limit check
Ongoing with low priority.
- Action(126) - Document the design of the workflow / production management components
Andrei will circulate a link for comments.
- Action(134) - Provide RunDB monitoring page
Ongoing.
- Action(135) - Send accounting from Online DIRAC instance via the gateway to PIC
Ongoing.
- Action(136) - Integrate SAM infrastructure tests
Ongoing.
2. Production Status Report
- Report from the Production meeting
Stuart mentioned that the two main points affecting operations are the NIKHEF CondDB access failures and the
inability to IN2P3 staging failures. Greig agreed to chase up the CondDB access failures with Marco Cl. Other relevant
discussions were regarding the possibility to ban jobs for different types of jobs e.g. simulation ok but not data
processing jobs. ( ACTION: Ricardo ). After some interesting discussion regarding how to cope with the reality of one
or more Tier1 sites not being available during a period of data taking it was agreed to make a proposal for how to
cope with this to settle on a policy ( ACTION: Andrei ).
- v4r9, v4r9p1 releases
The user upload default SE list was updated in the v4r9 release to be taken from an LHCb specific configuration
option.
- Future FEST weeks
Dates have been circulated by Olivier.
3. DIRAC Tasks
- Framework status
The request to have multiple DNs supported for users is implemented and will be tested by Andrei. All services and
agents are reporting their status for a current snapshot of the overall system.
- Bookkeeping status
Zoltan is absent.
- WMS status
Discussed above. New matching is being implemented. Database query times can be printed to the logs on demand.
- DMS status
Updates are being finalized, Andrew will update places in the code where the StorageElement is used directly
immediately prior to release.
- Production Management status
Alexey summarized the production request developments. Some relevant features have recently been implemented
by Zoltan for the BK.
- VO-boxes
The new machine will be available soon. Joel conveyed that storing of system log files were discussed at the recent
VO-box meeting. Andrei commented that the DIRAC system logging service nicely solves this problem for us.
4. AOB
There was some discussion regarding how to deal with problematic user jobs. Ricardo suggested (the other) Ricardo
develop tools and provide documentation for the analysis of job failures, this was welcomed.
Joel conveyed that any posters to be printed for CHEP should be ready by Tuesday of next week.
There are minutes attached to this event.
Show them.