Production and PASTE Meeting, 7th April 2009, CERN Chair: A.Tsaregorodtsev Minutes: S. Paterson CERN: P.Charpentier, Z.Mathe, S.Paterson, M.Rashid, R.Santinelli, M.Sapunov, A.C.Smith, F.Stagni, R.Vazquez Phone: R.Graciani, M.A.S.Miguelez, A.Tsaregorodtsev Apologies: J.Closier 0. Ongoing Action List (94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example) (95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test (96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings (109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (pending lhcbweb deployment) (111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation (116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC (122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots (124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side and protect data operations (126) - 20/01/09 - Andrei - Document the design of the workflow / production management components (132) - 03/03/09 - Stuart - Update LFN construction utility to store production output directories as a parameter (133) - 03/03/09 - Matvey - Provide request DB interface page (134) - 03/03/09 - Matvey,Andrew - Provide RunDB monitoring page (136) - 03/03/09 - Vladimir,Roberto,Joel - Integration of existing SAM infrastructure tests into DIRAC SAM framework (139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release (140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging... (141) - 07/04/09 - Vladimir,Roberto - Investigate reason for downtime calendar not being updated Completed Actions: Done - (135) - 03/03/09 - Andrew - Send accounting from Online DIRAC instance via the gateway to PIC Done - (137) - 10/03/09 - Ricardo - Implementing banning specific job types Done - (138) - 10/03/09 - Andrei - Make a proposal for coping with problematic Tier1 sites absent for long periods during data taking 1. Action Updates - Action(94) - SystemLoggingDB example Marcos was checking the FTS error messages and could not find the channel number, to be discussed offline. - Action(95),(96) - Service Monitoring Downtime messages were not propagated to the calendar. Philippe suggested that Robeto, Vladimir follow this up. ( ACTION: Robeto, Vladimir) - Action(109) - Expose DataLogging interface on the web page Ongoing. This is not yet deployed in production. - Action(111) - File merging productions To be discussed later. - Action(116) - CPU studies Ongoing. - Action(122) - Develop specialized SAM Agent to track and report errors with regard to pilots Ongoing. - Action(124) - Apply dataset limit check Ongoing with low priority. - Action(126) - Document the design of the workflow / production management components Ongoing. - Action(132) - Update LFN construction utility to store prod output directories as a parameter Just have to agree on a parameter to be stored, this is now possible after the recent production API developments. - Action(133) - RequestDB monitoring Comments to be sent to Matvey. - Action(134) - Provide RunDB monitoring page Ongoing. - Action(135) - Send accounting from Online DIRAC instance via the gateway to PIC Andrew installed the updated RunDB on the gateway. The online host should be able to send the accounting records. - Action(136) - Integrate SAM infrastructure tests Ongoing. - Action(137) - Implementing banning specific job types The job types are now part of the job matching mechanism, action closed. - Action(138) - Proposal for coping with problematic Tier1 sites absent for long periods during data taking Andrei updated the twiki page after last week's discussions. - Action(139) - gLexec testing after deployment at new sites No new deployment in production available for testing. - Action(140) - Banning faulty storage elements Ricardo will circulate the proposal for discussion, Andrew agreed to take this action on board. 2. Production Status Report - MC'09 For the MC generation 1) Gloria has generate 100 events with high number of pileup and found the CPU to increase linearly with this. 2) Marco is trying to reconstruct these events to find the total CPU time required to produce the events and the total event size. 3) A dummy production has been setup by Stuart (with an old version of Gauss) with the required workflow. Things remaining to do: - Wait for the release of the applications next (two) week. - Ensure the BK entry produced by the MC generation is invisible to users (nothing to do here) For the MC merging 4) SetupProject has been updated to allow 'SetupProject LHCb' to be performed. This requires a new version of install_project. 5) GaudiApplication was modified to allow the use of new configurables required for merging. Things remaining to do: - The new version of install_project which should be tested by Vladimir as soon as possible. - Ensure the BK entry produced by the MC generation is visible to production system. - Commission the merging workflow. - Determine the policy for removing the small DSTs after merging. - Data distribution policy to be configured for replication of the merged DSTs. - Update the BK report module to correctly report missing parameters. 3. DIRAC Tasks - Framework status Updating pylons to fix the security considerations recently raised by CERN-IT. - Bookkeeping status Production lookup and saving of file catalogs is now added to the BK system. - WMS status The VOMS server certs. were not correctly updated on the central lxplus Grid UI, Ricardo made an update of the TQ director to better react to this problem. The updated sandbox service is being prepared. - DMS status Mostly bug fixes. LFC admin interface was added to the client. - Production Management status Production request page should be tested after groups and new group properties are defined. - VO-boxes No update. - Web Portal Matvey mentioned that several pages are to be created for the data management. A refurbished site summary page is now available in production. - Services deployment ( hardware, configuration ) No response from the hardware request made over a month ago. 4. AOB The recent downtime cases will be investigated. Recent analysis jobs to ScotGrid are queued, this is no problem.