PASTE meeting

Tuesday 13 Jan 2009, 09:30 → 12:15 Europe/Zurich

2-R-014 (CERN)

Andrei Tsaregorodtsev

CERN, bat 2 R-014 To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202 To dial in to the conference: a. Dial +41 22 7676000 b. Enter access code 0165202

Production and PASTE Meeting, 13th January 2009, CERN Chair: A.Tsaregorodtsev Minutes: S. Paterson CERN: J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith Phone: N.Brook, M.Sapunov, A.Tsaregorodtsev Apologies: P.Charpentier 0. Ongoing Action List (07) - 27/11/07 - Zoltan - Ongoing new bookkeeping system development. BK dataset queries. (57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes (83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification) (90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs (93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB (94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test (96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings (106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install (107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT (109) - 14/09/08 - Matvey,Andrew - Expose DataLogging interface on the web page (111) - 14/09/08 - Philippe,Alexei,Andrei - Productions with file merging and associated data distribution implementation (113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties (115) - 21/10/08 - Vladimir,Matvey - Update SAM Agent to monitor and manage submitted jobs and publish results for faulty CEs to the SAM DB (116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC (117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director (120) - 16/12/08 - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (121) - 16/12/08 - Andrei, Alexey - Add production extensions to logging information (pending release) (122) - 13/01/09 - Ricardo - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots (123) - 13/01/09 - Joel - Clean up VOMSRS after agreeing on a policy of one DN per user (124) - 13/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) Completed Actions: 1. Action Updates - Action(07) - Prototype Bookkeeping Zoltan is preparing the new methods for automating the processing pass insertion and will require a new release of DIRAC for client and server. - Action(57) - Automatic publishing of BDII / GOCDB information in the CS Roberto and Vladimir will prepare a summary of the outstanding items relating to this action point. - Action(83) - gLexec tests Nothing new to report, SCAS is still under certification. There has not been a great deal of progress regarding the deployment of the pilot role. - Action(90) - Production Output Data parameter Ongoing. - Action(93) - Improve RAW data monitoring tools and status of files in online runDB Ongoing. - Action(94) - SystemLoggingDB example Andrew has the code for categorizing the errors, this should be entered into the Logging system so this will be refreshed with Marcos. - Action(95),(96) - Service Monitoring There were no results for some SAM tests over the Christmas break due to WMS unavailability over Christmas. Stuart mentioned there would be added protection if these tests were moved into the DIRAC SAM framework since multiple WMS instances and multiple pilots can be submitted in this case. There was some discussion regarding the policy with regard to jobs not starting, it was agreed to report test jobs not starting as a failure with sufficient information about attempted submissions. This implies a decoupling of the SAM pilot submissions with a dedicated director that can publish errors from the WMS instances. - Action(106) - Ricardo - Produce twiki documentation for dirac-install Ongoing. - Action(107) - Produce proposal for how to perform standardised installation using install_project/CMT The updated installation procedure is tested with v4r4 and several environments. Recent problems were reported with the version of DIRAC used by Ganga (v4r3p1), Joel commented that Will is testing v4r4. - Action(109) - Expose DataLogging interface on the web page Ongoing. - Action(111) - File merging productions Ongoing. - Action(113) - CS Documentation Ongoing. - Action(115) - SAM Agent Vladimir's agent is running happily (on volhcb02) and should be committed to CVS and the installation script should be updated with this LHCbSystem agent. - Action(116) - CPU studies Ongoing. - Action(117) - Generic pilot site attribute Ongoing. - Action(120) - Update TQ Director to deal with a mixture of generic / private pilots at sites Ongoing. - Action(121) - Add production extensions to logging information Just pending the next DIRAC release. 2. Production Status Report - Report from the Production meeting: Stuart mentioned that 14.8k concurrent jobs were running during last weekend. The fix for stale NFS file handles at sites should be available in due course. Persistent problems have been observed at CNAF due to the shared area access, Roberto agreed to contact the site administrators. As a reminder the list of ongoing FEST preparations is below: - BK query population of datasets - Finalize derived production mechanism - Optimizations for the creation of jobs (for productions with input data) - Expose production priority boosting mechanism via production API - Introduce refined status machine to allow archiving of productions - Introduce a ProductionGroup parameter to improve monitoring (analogous to JobGroup). Andrei agreed to circulate proposed dates for a DIRAC week, prior to FEST. - Analysis of the Xmas and recent runs DIRAC was running in an unattended mode for 2 weeks. Certain limitations on our side were highlighted, resulting in the following recommendations: - Bulk job submission for production and user jobs - Tuning of pilot submissions (see below) - Memory consumption of JobStateUpdate is a limiting factor - Optimizations in job wrapper and on the service side via decoupling the heartbeats - Pilot summary page is a good indicator of how well a site is functioning, this should be tuned - Simple plots such as in DIRAC2 should be made available in DIRAC3 - Timeleft utility would be useful for preventing stalled jobs - New WMS instance Can be added to the list of gLite WMS instances for the submission pool. - Job exceeding CPU limit at IN2P3 The CPU limit at IN2P3 needed to be increased and Andrei will chase this up with the local admins. After some discussion it was agreed to only allow one active DN for the VO per user since the only use-case for two DNs was changing CA. This means that VOMSRS can be cleaned up for the cases where users have multiple DNs in the VO. ( ACTION: Joel ) 3. DIRAC3 Tasks - Optimization of pilot submission and status evaluation are summarized above - Decoupling SAM job submissions - If pilots for certain queues are failing they should not be retried too often - Tuning for parametric jobs, taking WMS occupancy into account - Using generic pilots at T1 sites Ongoing. - Readiness for user analysis in DIRAC3 ( bookkeeping, priorities, jobs with large input, etc ) For jobs with large input data sets we recently noticed the JDL was incomplete, Stuart will apply a limit for user jobs. ( ACTION: Stuart ) From last week: visualization of Task Queues on the monitoring portal would also be advantageous e.g. TQ requirements, summary of pilots, number of jobs etc. 4. AOB None

09:30 → 09:50 Ongoing actions

(07) - 27/11/07 - Zoltan - Ongoing new bookkeeping system development. BK dataset queries. (57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes (83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification) (90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs (93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online runDB (94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test (96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization (106) - 30/09/08 - Ricardo - Produce twiki documentation for dirac-install (107) - 30/09/08 - Ricardo,Philippe,Joel,Hubert,MarcoC - Produce proposal for how to perform standardised installation using install_project/CMT (109) - 14/09/08 - Matvey,Andrew - Expose DataLogging interface on the web page (111) - 14/09/08 - Philippe,Alexei,Andrei - Productions with file merging and associated data distribution implementation (113) - 21/10/08 - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties (115) - 21/10/08 - Vladimir,Matvey - Update SAM Agent to monitor and manage submitted jobs and publish results for faulty CEs to the SAM DB (116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC (117) - 18/11/08 - Ricardo - Add a site attribute to reflect whether generic pilots are accepted there, update the Director (120) - 16/12/08 - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (121) - 16/12/08 - Andrei, Alexey - Add production extensions to logging information (pending release)

09:50 → 10:05 Production status report

- Report from the Production meeting: - Analysis of the Xmas and recent runs - New WMS instance - Job exceeding CPU limit

10:05 → 10:35 DIRAC3 tasks

- Optimization of pilot submission and status evaluation - Using generic pilots at T1 sites - Readiness for user analysis in DIRAC3 ( bookkeeping, priorities, jobs with large input, etc )

10:35 → 10:45 AOB