Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 3rd February 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith
Phone: R.Graciani, A.Tsaregorodtsev, A.Zhelezov
Apologies: M.Sapunov
0. Ongoing Action List
(57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes
(83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification)
(90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs
(93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB
(94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test
(96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings
(106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install
(109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (pending lhcbweb deployment)
(111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation
(113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties
(116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC
(117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director
(120) - 16/12/08 - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (and document the submission policy)
(122) - 13/01/09 - Ricardo - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots
(124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side e.g. LHCbJob
(125) - 20/01/09 - Alexey - Production request GUI progress
(126) - 20/01/09 - Andrei - Document the design of the workflow / production management components
(127) - 20/01/09 - Andrei,Adri - Multiple DNs in DIRAC for use-case of DN change (check job policies and web portal policies)
(128) - 27/01/09 - Adri - Add failover requests to the accounting and monitoring clients
(129) - 27/01/09 - Ricardo - Document the job prioritization mechanism
(130) - 03/02/09 - Philippe,Andrew,Stuart - Prepare proposal for the DIRAC release procedure
(131) - 03/02/09 - Andrei, Stuart - Test BK production
(132) - 03/02/09 - Ricardo - Provide example of stalling Gauss jobs to Gloria
Completed Actions:
Moved to DIRAC Tasks - (07) - 27/11/07 - Zoltan - Ongoing new bookkeeping system development. BK dataset queries.
Done - (107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT
1. Action Updates
- Action(07) - Prototype Bookkeeping
Zoltan has started on the new methods required to express the Data Quality flags. Agreed to close this action point
and have a regular summary in the DIRAC tasks.
- Action(57) - Automatic publishing of BDII / GOCDB information in the CS
Ongoing. Andrei commented that any changes of site name should be reflected.
- Action(83) - gLexec tests
Roberto commented that a suggestion was made to the PPS people to deploy gLexec on some Tier-1 sites for testing.
Andrei agreed to check whether this prototype deployment is feasible for IN2P3.
- Action(90) - Production Output Data parameter
Ongoing.
- Action(93) - Improve RAW data monitoring tools and status of files in online runDB
Rashid and Andrew have been working on a service to use the Online RunDB interface to check the status of files.
- Action(94) - SystemLoggingDB example
Ongoing.
- Action(95),(96) - Service Monitoring
A recently failing test at CNAF has highlighted some misleading errors that will be better trapped in the future.
Displaying the DIRAC site names on the Dashboard is ongoing. Andrei agreed with Pablo and Elisa the format to
convey the current site mask. After some discussion it was agreed to convey the status of the site as reflected in the
site mask.
- Action(106) - Ricardo - Produce twiki documentation for dirac-install
Ongoing.
- Action(107) - Produce proposal for how to perform standardised installation using install_project/CMT
There was some discussion regarding the availability of LCG middleware on macs. It was agreed to build a binary
release of DIRAC for the mac platform and Adri / Ricardo will try to do this.
- Action(109) - Expose DataLogging interface on the web page
Ongoing.
- Action(111) - File merging productions
Ongoing.
- Action(113) - CS Documentation
Ongoing.
- Action(116) - CPU studies
Ongoing.
- Action(117) - Generic pilot site attribute
Ongoing.
- Action(120) - Update TQ Director to deal with a mixture of generic / private pilots at sites
Ricardo will provide documentation for this.
- Action(121) - Add production extensions to logging information
Released in DIRAC v4r5.
- Action(122) - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots
Ricardo commented that tracking errors by WMS is not Andrei and Stuart added to this action point to come up with
a proposal together with Ricardo.
- Action(124) - Apply dataset limit check
Ongoing with low priority.
- Action(125) - Production Request GUI
Alexey is almost finished the database part of the request GUI and will try to integrate this new service with the
central machines.
- Action(126) - Document the design of the workflow / production management components
Ongoing.
- Action(127) - Investigate feasibility of supporting multiple DNs in DIRAC only for use-case of DN change
Ricardo conveyed that this is feasible for everything except the monitoring portal where Matvey / Adri should
comment. A few fixes are pending in the JobPolicy object.
- Action(128) - Add failover requests to the accounting and monitoring clients
Ongoing.
- Action(129) - Document the job prioritization mechanism
Ongoing.
2. Production Status Report
- Report from the Production meeting
The dummy activity is ongoing, recent Generator requests are running smoothly. Urgently need to clean production
monitoring page. Still pending the fix for stale NFS file handles from core software. The most pressing operational
issues are file access at IN2P3 and the shared area at CNAF.
- DIRAC release procedure
Philippe, Andrew and Stuart will come up with a proposal for the DIRAC release procedure. ( ACTION: Philippe, Andrew, Stuart )
- BK query population of datasets
Andrei mentioned that the code is prepared for this and can be tested in a clean way in production. ( ACTION: Andrei, Stuart )
As a reminder the list of ongoing FEST developments is below:
- Finalize derived production mechanism
- Optimizations for the creation of jobs (for productions with input data)
- Expose production priority boosting mechanism via production API
- Introduce refined status machine to allow archiving of productions
- Introduce a ProductionGroup parameter to improve monitoring (analogous to JobGroup).
3. DIRAC (Renamed from DIRAC3 :) Tasks
The progress of the BK web GUI was queried and Andrei agreed to circulate the URL of this page.
Ricardo raised the issue of recent stalling jobs in Gauss and agreed to raise this with Gloria, the example he has is
reproducible on the local machine. ( ACTION: Ricardo )
Andrew mentioned that the cleaning of the LogSE should include the migration of log files to Castor for productions as
soon as they are marked as finished.
Reminder of Andrei's summary of the outstanding tasks after the Christmas post-mortem:
- Bulk job submission for production and user jobs
- Tuning of pilot submissions (see below)
- Memory consumption of JobStateUpdate is a limiting factor
- Optimizations in job wrapper and on the service side via decoupling the heartbeats
- Pilot summary page is a good indicator of how well a site is functioning, this should be tuned
- Simple plots such as in DIRAC2 should be made available in DIRAC3
- Timeleft utility would be useful for preventing stalled jobs
4. AOB
Philippe wondered whether T-shirts / sweatshirts would be desirable for CHEP and agreed to put up a Doodle poll.
CHEP is 7 weeks away so those with contributions are advised to make a start. Meeting closed at 11:36.
There are minutes attached to this event.
Show them.