PASTE meeting

Tuesday 23 Jun 2009, 09:30 → 13:15 Europe/Zurich

2-R-014 (CERN)

Andrei Tsaregorodtsev

CERN, bat 2 R-014 To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202 To dial in to the conference: a. Dial +41 22 7676000 b. Enter access code 0165202

Production and PASTE Meeting, 23th June 2009, CERN Chair: A.Tsaregorodtsev Minutes: A.Tsaregorodtsev CERN: Z.Mathe,F.Stagni,J.Closier,A.C.Smith,V.Romanovsky,R.Mamunur Phone: A.Zhelezov, A.Tsaregorodtsev 0. Ongoing Action List (94) - 23/09/08 - Andrew,Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (111) - 14/09/08 - Alexei,Andrei - Production request page, file merging and associated data distribution implementation (116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units (122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots (133) - 03/03/09 - Matvey - Provide request DB interface page (139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release (140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging... (146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification. Should be complemented with the installed SW monitoring tools. (151) - 26/05/09 - Andrew,Andrei - Update central components with optimized BK methods (152) - 26/05/09 - Stuart - Investigate the crashes when using the copy local option for SQL DDDB files (156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures (157) - 16/06/09 - Andrei,Ricardo - Prepare plan with timestamps for obtaining new hardware for production system (159) - 16/06/09 - Philippe - Investigate the issue of persisting production log files with the PPG Actions closed: (153) - 26/05/09 - Ricardo,Joel - Investigate optimal hardware configurations for DIRAC servers continued in Action 157 (158) - 16/06/09 - Andrei - Correct the stalled job reporting on the production monitoring page 1. Action Updates - Action(94) - Logging system usage examples Ongoing. - Action(111) - Production definition tools Alexei reported that minor suggestions made by Philippe after the initial testing are being implemented. The outcome of the Production Request procedure will be a script which allows to generate the specified production. The template for this script will be derived from /DIRAC/ProductionManagementSystem/scripts/dirac-production-MC09-create.py script used now for MC'09 production definitions. Alexei agreed to provide the template and to test the resulting production scripts. - Action(116) - CPU normalization studies, Filling mode The filling mode with the CPU left estimate is switched on now and the number of job per pilot in the recent days is above 1.0 (~1.1). This is a good sign. However, jobs should specify a more realistic MaxCPUTime requirements in order to take advantage of that. This is not the case for the current MC'09 production jobs. - Action(122) - Analysis of the SAM pilot submission errors. Ongoing. - Action(133) - RequestDB monitoring Ongoing. - Action(139) - Ongoing. - Action(140) - Banning faulty storage elements Ongoing. - Action(146) - SW installation framework using CS data Ongoing. - Action(147) - new DIRAC service host The host OS should be updated to SLC4. The machine is expected this week. - Action(148) - SVN/Trac server Ongoing. SW shared area celaning procedure is done. - Action(151) - Update central components with optimized BK methods Implemented, being tested, will be included in v4r15 at lease for the WMS part - Action(152) - Investigate the crashes when using the copy local option for SQL DDDB files To be confirmed by testing the fixes done by Marco - Action(156) - Refine VO-box setup to minimize impact of updates and maintenance procedures Ongoing - Action(156) - Plan for new hardware for production system See document attached to the agenda. Everybody is invited to comment. - Action(159) - Investigate the issue of persisting production log files with the PPG Ongoing 2. Production issues MC'09 production is running smoothly, the total of 390000 are created. It was agreed to stop new job submission to let people check if everything is OK. File merging for XDST production nu=1,2,3,4 is done. The resulting file sizes are up to 27 GB. The large file size results in a large upload times which provoked one job to be killed by the watchdog as stalled. Smaller file sizes are more desirable. File grouping by size rather than by number of files is necessary. Some sites have a large number of pilots in a local batch system waiting. This slows down the further pilot submissions which considers that enough pilots are already in the system. This logic improvement is needed. Many production jobs are triggering submission of the provate pilots. Those can not take user jobs. This results in th fact that user jobs are penalized and their priority can not be properly applied. Migration of the production service from volhcb09 to volhcb13 should happen this week. This will be done with the v4r15 of DIRAC which is being prepared now. Zoltan will move the development BK system from volhcb04 to volhcb05 host. This will free the volhcb04 for the services now running now on volhcb01. This migration will also be accomplished this work and volhcb01 will be discontinued. Joel also reported that the new volhcb14 file server is in production now. This will be used for studying of the separate MySQL server installation. 3. DIRAC Tasks - Infrastructure monitoring (SAM,SLS,Dashboard) Joel reported that a not describing the LHCb SAM procedures is in preparation. Joel and Roberto participated to the tutorial for Nagios based service monitoring system. The system will necessitate experiment specific servers for which specifications are not yet clear. The Nagios based tools are not likely to be finalized by September. - Framework status - Bookkeeping status Zoltan has provided the time stamps for each file marking its last modification time. This will be available in v4r15. - WMS status New Sandbox service and new Pilot Director mechanisms will be released with v4r15. - DMS Andrew reported that he is working on the update of the Stager service. It will use the full SRM staging request info rather than querying file by file. The goal is to avoid current problems when two jobs are asking to stage the same file and one of them can be stuck. The new service will provide also file pinning and disk cache management capabilities. The Stager system agents will provide data integrity info to spot problems before the correponding jobs get submitted. - Production Management status See Action 111 - VO-boxes No update - Web Portal No update - Services deployment ( hardware, configuration ) See the document attached to the agenda - Documentation No update 4. AOB

09:50 → 10:10 Ongoing actions

(94) - 23/09/08 - Andrew,Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (111) - 14/09/08 - Alexei,Andrei - Production request page, file merging and associated data distribution implementation (116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units (122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots (133) - 03/03/09 - Matvey - Provide request DB interface page (139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release (140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging... (146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification. Should be complemented with the installed SW monitoring tools. (151) - 26/05/09 - Andrew,Andrei - Update central components with optimized BK methods (152) - 26/05/09 - Stuart - Investigate the crashes when using the copy local option for SQL DDDB files (153) - 26/05/09 - Ricardo,Joel - Investigate optimal hardware configurations for DIRAC servers (156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures (157) - 16/06/09 - Andrei,Ricardo - Prepare plan with timestamps for obtaining new hardware for production system (158) - 16/06/09 - Andrei - Correct the stalled job reporting on the production monitoring page (159) - 16/06/09 - Philippe - Investigate the issue of persisting production log files with the PPG

document DIRAC_LHCb-Production_Setup.pdf

10:10 → 10:25 Production status report

- Report from the Production meeting - v4r15 release - volhcb13 migration

10:25 → 10:55 DIRAC tasks

- Infrastructure monitoring (SAM,SLS,Dashboard) - Framework status - Bookkeeping status - WMS status - DMS status - Production Management status - VO-boxes - Web Portal - Services deployment ( hardware, configuration ) - Documentation

10:55 → 11:05 AOB