PASTE meeting

Tuesday 27 Jan 2009, 09:30 → 12:15 Europe/Zurich

2-R-014 (CERN)

Andrei Tsaregorodtsev

CERN, bat 2 R-014 To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202 To dial in to the conference: a. Dial +41 22 7676000 b. Enter access code 0165202

Production and PASTE Meeting, 27th January 2009, CERN Chair: A.Tsaregorodtsev Minutes: S. Paterson CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith, A.Zhelezov Phone: R.Graciani, A.Tsaregorodtsev Apologies: M.Sapunov 0. Ongoing Action List (07) - 27/11/07 - Zoltan - Ongoing new bookkeeping system development. BK dataset queries. (57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes (83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification) (90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs (93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB (94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test (96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings (106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install (107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT (109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (111) - 14/09/08 - Philippe,Alexei,Andrei - Productions with file merging and associated data distribution implementation (113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties (116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC (117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director (120) - 16/12/08 - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (and document the submission policy) (122) - 13/01/09 - Ricardo - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots (124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side e.g. LHCbJob (125) - 20/01/09 - Alexey - Production request GUI progress (126) - 20/01/09 - Andrei - Document the design of the workflow / production management components (127) - 20/01/09 - Ricardo - Investigate feasibility of supporting multiple DNs in DIRAC only for use-case of DN change (128) - 27/01/09 - Adri - Add failover requests to the accounting and monitoring clients (129) - 27/01/09 - Ricardo - Document the job prioritization mechanism Completed Actions: Done - (121) - 16/12/08 - Andrei, Alexey - Add production extensions to logging information (pending release) Done - (123) - 13/01/09 - Joel - VOMS(RS) consistency 1. Action Updates - Action(07) - Prototype Bookkeeping Zoltan summarized the new BK features. The fill and run numbers were added to the job table for the online, the XML reports are translated correctly to these DB fields. There were several GUI updates including ancestor files, log files and processing pass information for given files. The production processing pass table has been reorganised, methods to check whether the processing pass already exists and to insert these are available. Stuart asked whether a given processing pass name could be augmented with other compatible project versions and this is now possible. The data quality flags are pending but will be part of the next release. - Action(57) - Automatic publishing of BDII / GOCDB information in the CS For each site the CE names are defined for DIRAC sites, additional information farmed from the BDII has been inserted there in case it is useful. Having an agent to automate the maintenance of CEs would certainly be desirable, the existing code will be formulated as an agent in the coming weeks. - Action(83) - gLexec tests Nothing new to report, SCAS is still under certification. - Action(90) - Production Output Data parameter Ongoing. - Action(93) - Improve RAW data monitoring tools and status of files in online runDB Ongoing. - Action(94) - SystemLoggingDB example Ongoing. - Action(95),(96) - Service Monitoring SAM SLS development is ongoing. Andrew asked why the DMS unit test was removed from the list of critical tests. Roberto noticed this had stopped working and will continue to monitor the situation. File access tests have spotted incompatibilities with pure 64bit OS running in 32bit compatibility mode. - Action(106) - Ricardo - Produce twiki documentation for dirac-install Ongoing. - Action(107) - Produce proposal for how to perform standardised installation using install_project/CMT Some developments have been pending the next release of DIRAC. - Action(109) - Expose DataLogging interface on the web page Ongoing. - Action(111) - File merging productions Ongoing. - Action(113) - CS Documentation Ongoing. - Action(116) - CPU studies Ongoing. - Action(117) - Generic pilot site attribute Ongoing. - Action(120) - Update TQ Director to deal with a mixture of generic / private pilots at sites Ongoing. It was agreed to proceed with a fraction of private / generic pilot submissions for sites using the provision or otherwise of the LHCb pilot role as a hook. Ricardo agreed to document the policy. - Action(121) - Add production extensions to logging information Released in DIRAC v4r5. - Action(122) - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots After some discussion it was decided to proceed with writing a SAM pilot monitoring agent that can publish the cause of failures to the SAM DB. Ricardo confirmed the separation of SAM pilot submissions from production pilot submissions is ongoing and under testing. After the recent operational experience of low throughput, Ricardo was asked to check the cause but it looks very much like consistent list-match failures for the WMS instances. - Action(124) - Apply dataset limit check Ongoing. - Action(125) - Production Request GUI Alexey will work with Stuart this week to finalize the prototype. - Action(126) - Document the design of the workflow / production management components Ongoing. - Action(127) - Investigate feasibility of supporting multiple DNs in DIRAC only for use-case of DN change Ricardo conveyed that this is feasible for everything except the monitoring portal where Matvey / Adri should comment. A few fixes are pending in the JobPolicy object. 2. Production Status Report - Report from the Production meeting User file GUIDs problem will be resolved by the end of the day. The dummy activity is ongoing, new DC09 requests made (software being installed). Urgently need to clean production monitoring page. Still pending the fix for stale NFS file handles from core software. The most pressing operational issue was the shared area at CNAF being unusable. Andrew raised the point of failover for accounting and monitoring reports and it was agreed to add this to the accounting client. ( ACTION: Adri ) - v4r6 release Andrei is ready to make a release of DIRAC and will proceed with this during the day. - Pilot submission See above discussions. - FEST info As a reminder the list of ongoing FEST preparations is below: - BK query population of datasets - Finalize derived production mechanism - Optimizations for the creation of jobs (for productions with input data) - Expose production priority boosting mechanism via production API - Introduce refined status machine to allow archiving of productions - Introduce a ProductionGroup parameter to improve monitoring (analogous to JobGroup). 3. DIRAC3 Tasks - Production definitions, Bk queries Several updates are pending the v4r6 release. - Task prioritization Ricardo will summarize the prioritization mechanism on a twiki page after the discussions during the meeting. ( ACTION: Ricardo ) Reminder of Andrei's summary of the outstanding tasks after the Christmas post-mortem: - Bulk job submission for production and user jobs - Tuning of pilot submissions (see below) - Memory consumption of JobStateUpdate is a limiting factor - Optimizations in job wrapper and on the service side via decoupling the heartbeats - Pilot summary page is a good indicator of how well a site is functioning, this should be tuned - Simple plots such as in DIRAC2 should be made available in DIRAC3 - Timeleft utility would be useful for preventing stalled jobs 4. AOB Meeting closed at 12:28.

09:50 → 10:10 Ongoing actions

(07) - 27/11/07 - Zoltan - Ongoing new bookkeeping system development. BK dataset queries. (57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes (83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification) (90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs (93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB (94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test (96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings (106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install (107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT (109) - 14/09/08 - Matvey,Andrew - Expose DataLogging interface on the web page (111) - 14/09/08 - Philippe,Alexei,Andrei - Productions with file merging and associated data distribution implementation (113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties (116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC (117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director (120) - 16/12/08 - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (121) - 16/12/08 - Andrei, Alexey - Add production extensions to logging information (pending release) (122) - 13/01/09 - Ricardo - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots (123) - 13/01/09 - Joel - Clean up VOMS(RS) to ensure all users have consistent roles in VOMS (124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side e.g. LHCbJob (125) - 20/01/09 - Alexey - Production request GUI progress (126) - 20/01/09 - Andrei - Document the design of the workflow / production management components (127) - 20/01/09 - Ricardo - Investigate feasibility of supporting multiple DNs in DIRAC only for use-case of DN change

10:10 → 10:25 Production status report

- Report from the Production meeting: - v4r6 release - Pilot submission - FEST info

10:25 → 10:55 DIRAC3 tasks

- Status of production tools: - Production Request GUI, - Production definitions, Bk queries - Task prioritization - Bk Web GUI

10:55 → 11:05 AOB