PASTE meeting

Tuesday 26 Feb 2008, 09:30 → 12:00 Europe/Zurich

Andrei Tsaregorodtsev

CERN, bat 2 R-030 To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882 To dial in to the conference: a. Dial +41 22 7676000 b. Enter access code 0122882

Production and PASTE Meeting, 26th February 2008, CERN Chair: A.Tsaregorodtsev Minutes: S. Paterson CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, Z.Mathe, S.Paterson, R.Santinelli Phone: R.Bernet, N.Brook, R.Graciani, G.Kuznetszov, M.Sapunov, U.Marconi, A.Tsaregorodtsev, A.Zhelezov Apologies: A.Casajus, M.A.S.Miguelez, R.Nandakumar, A.C.Smith 0. Ongoing Action List (01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this (04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions (07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07 (09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room (25) - 08/01/08 - Marianne - Resolve why some files are double-counted in Bookkeeping (27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4 (36) - 12/02/08 - Roberto, Gianluca - Investigate other experiments usage and functionality of the Dashboard (38) - 26/02/08 - Zoltan, Elisa, Marianne - Review the view-building process in the Bookkeeping (39) - 26/02/08 - Joel - Test DIRAC v2r19 release before deployment (40) - 26/02/08 - Gianluca - Investigate causes of many aborted pilot agents by the LCG RBs (41) - 26/02/08 - Marianne - Trigger replication of stripped data to other Tier-1s (RAL) (00) - 26/02/08 - DIRAC developers - DIRAC3 (ongoing) Completed Actions Done - (03) - 30/10/07 - Marianne - Update of LFC replica flags (pending NIKHEF) Done - (32) - 22/01/08 - Gianluca, Marianne - Identify and resolve faulty productions with duplicated events Done - (35) - 12/02/08 - Zoltan, Andrei - Setup DIRAC3 BK interface on volhcb05 (install DIRAC3 there) Done - (37) - 12/02/08 - Andrei - Update proxy handling infrastructure in DIRAC2 1. Production Status Report Minutes from the production meeting are available here. - Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01) Gianluca reported that the agent working on the maximum reschedulings jobs is running but something is not clear, when agent spots a job and sets a file to unused, the jobID in the file is not cleared. Gianluca will contact Gennady to understand this. The agents to work from the Integrity DB and perform corrective actions is pending. ( ACTION: Gianluca ) - RAL and PIC migrations - Action(04) The current pace of 1 tape per day has continued at RAL. The RAL-disk definition has now been changed to Castor but old VO-box transfers have found to contain the port number in the path. Marianne confirmed that PIC-tape LFC entries are now being removed. - Bookkeeping prototype - Action(07) Zoltan reported that the Bookkeeping Manager is being ported to DIRAC3, this is responsible for processing the XML. Work has also been started on the DataManager that puts the XML into the database. After this is completed, the Java part of this can be retired. Elisa has circulated a proposal for the update of the Bookkeeping database schema and this is under discussion. - LHCb SAM framework for SE / SRM tests - Action (09) Roberto conveyed that several tests were in place but there were problems during publication to the SAM DB. It was agreed to transform this action to include a general service monitoring framework. The main goal is to design a virtual control room. - Missing number of events in BK - Action(25) Marianne thinks this is related to the creation of the views but investigations are ongoing. Philippe remarked that this is misleading for users and should be pursued. It appears that for stripped data, ancestors were inserted for multiple versions of Brunel but this behaviour is clearly not desired. - Review of view-building in the BK - Action(38) This activity was made into a separate action. - DIRAC v2r19 release Joel is in the process of testing the v2r19 release that will be installed on Production and User systems. ( ACTION: Joel ) - Gianluca raised the issue of pilots being aborted by the LCG RBs, this should be investigated ( ACTION: Gianluca ) - Due to the disk space constraints at the Tier-1s, stripped data needed by the physicists (and supposed to be at all Tier-1s in the Computing Model) is not there. Since RAL now has increased space it was agreed to identify the stripped data from the Bookkeeping and trigger the replication there. ( ACTION: Marianne ) 2. CCRC08 Operations Issues - Data transfer status IN2P3 disk cache space parameter was set to a low value causing an outage for the disk transfers, after this was reset things started to go smoothly. Data transfers and running jobs at CNAF stopped due to not having enough LSF slots but this has now been increased. Andrew entered an eLog entry mentioning 5 files terminally failing to transfer at NIKHEF, most likely due to a similar space parameter issue. - Production Management status Defining a production with the run numbers was discussed yesterday and a more convenient solution should be pursued. Gennady mentioned that the filter for the production should be wrapped to trap any errors. - Application execution failures The suspicion is that Root does not understand the end of the file causing a final error in the application log. Another issue is that it seems dCache sites are resetting their gsidcap doors fairly regularly. Stuart mentioned that one of the positives from his exercise is that most errors (also successes) can now be trapped in the workflow and meaningfully displayed on the webpage. - Services/framework failures Andrei conveyed that the problems of proxy retrieval (relating to the VOMS client) should now be solved. Several clients have been observing timeouts / connections being refused almost at random, it is a suspected problem in the framework that Adri is currently investigating. As a temporary measure, e.g. the BK should be retried several times. Adri corrected a problem with services hanging but this subsequently revealed a memory leak on further inspection by Ricardo. - Monitoring status The heartbeat time and production monitor have now been conveyed to lhcbweb.pic.es, Matvey is continuing to improve the interface. It is now possible to start/stop the productions via the interface. The job history plot groupings will be changed to allow more of a comparison between sites. The decision about resuming CCRC runs will be taken at today's 5pm meeting. 3. DIRAC3 Tasks - volhcb01 status Ricardo mentioned that volhcb01 is ready to be used but we are waiting for the DBs and users to be configured. The list of necessary ports are ready to be conveyed to IT for opening. - Log SE and Failover status Developments are ongoing. 4. AOB None

09:30 → 09:50 Ongoing actions

(01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this (merged with Action(02)) (03) - 30/10/07 - Marianne - Update of LFC replica flags (pending NIKHEF) (04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions (07) - 27/11/07 - Elisa, Marianne - Prototype bookkeeping with API wrapped in DISET service after ensuring AMGA and Oracle availability on volhcb07 (09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop LHCb SAM framework and summarize architecture for SE tests (25) - 08/01/08 - Marianne - Resolve why number of processed events is missing in Bookkeeping (also cause of missing files) (27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4 (32) - 22/01/08 - Gianluca, Marianne - Identify and resolve faulty productions with duplicated events (35) - 12/02/08 - Zoltan, Andrei - Setup DIRAC3 BK interface on volhcb05 (install DIRAC3 there) (36) - 12/02/08 - Roberto, Gianluca - Investigate other experiments usage and functionality of the Dashboard (37) - 12/02/08 - Andrei - Update proxy handling infrastructure in DIRAC2 (00) - 12/02/08 - DIRAC developers - DIRAC3 (ongoing)

09:50 → 10:05 Production status report

- Report from the Production meeting:

10:05 → 10:35 CCRC operations issues

- Data transfer status - Production Management status - Application execution failures - Services/framework failures - Monitoring status

10:35 → 11:05 DIRAC3 tasks

- volhcb01 status - Log SE status - Failover status

11:05 → 11:15 AOB