Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122929
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 4th August 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: S.Paterson
CERN: P.Charpentier, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith, F.Stagni
Phone: V.Fernandez, R.Graciani, M.A.S.Miguelez, R.Nandakumar, M.Sapunov, A.Tsaregorodtsev, A.Zhelezov
Apologies: J.Closier
0. Ongoing Action List
(94) - 23/09/08 - Andrew,Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(133) - 03/03/09 - Matvey - Provide request DB interface page
(139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release
(140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging...
(146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification. Should be complemented with the installed SW monitoring tools.
(151) - 26/05/09 - Andrew - Update central components with optimized BK methods using file time stamps
(161) - 07/07/09 - Alexei - define Production Request for ongoing production to allow its monitoring
(162) - 07/07/09 - Ricardo,Andrei - document the TQ priorities algorithm
(163) - 07/07/09 - Joel - investigate possible use of CERN backup tools for log files archiving.
(164) - 16/07/09 - Matvey,Adri - Formalize and simplify the release procedure for new web portal pages (including regular DIRAC version updates)
(165) - 16/07/09 - Ricardo - Add WMS accounting for visualization of 'time to running' and other accounting metrics
(166) - 16/07/09 - Ricardo - Set up volhcb05 as the new certification machine
(167) - 16/07/09 - Philippe,Ricardo,Roberto - Ensure new middleware bindings are available and start SLC5 commissioning on volhcb12
(168) - 04/08/09 - Matvey,Alexey - Release procedure for the web portal, ensure request page is in production
(169) - 04/08/09 - Stuart,Vladimir - Continue SLC5 testing at the sites where this is available
Completed Actions:
Done - (111) - 14/09/08 - Alexei,Andrei - Production request page, file merging and associated data distribution implementation
Done - (156) - 02/06/09 - Ricardo,Joel - Refine VO-box setup to minimize impact of updates and maintenance procedures
1. Action Updates
- Action(94) - SystemLoggingDB example
Ongoing, Marcos will make a tool available in CVS for this.
- Action(111) - File merging productions & request page
Stuart summarized that production templates for Reconstruction, MC09 and MC09 merging are in place. These
automatically associate productions to requests and start them etc. The major required development would be to
release this in the production web portal. It was noted that the pace of updates to the web portal did not match the
releases of DIRAC, this procedure will be refined ( ACTION: Matvey, Alexey ). Adding the PDF to a twiki such that a
'help' button from the portal links there would be very beneficial. Philippe mentioned that the representation of
numbers in the request page would be greatly improved with comma separated digits and Alexey agreed to put this in
place.
- Action(116) - CPU studies
Ongoing.
- Action(122) - Develop specialized SAM Agent to track and report errors with regard to pilots
Ongoing.
- Action(133) - RequestDB monitoring
Matvey will circulate the link to the request summary page on lhcbtest for comments. Andrei will make a proposal on the restructuring of the portal menus.
- Action(139) - gLexec testing after deployment at new sites
The pilot exercise has been closed, Roberto reported some conclusions to Antonio. Still the deployment /
configuration of glexec SCAS is not trivial. The aim of the second iteration of tests was to demonstrate that the
environment saving script was working but this was not the case when tested at some sites until configuration issues
were used. Other configuration problems at IN2P3 prevented even to run tests there. For the LHCb official statement
it was agreed to convey that we could only run on the correctly configured sites with IN2P3 being a notable exception.
- Action(140) - Banning faulty storage elements
Ongoing, technically completed but Andrew will add scripts to streamline this process and is starting on the
documentation.
- Action(146) - SAM software distribution synchronization with the CS
The new module was added to the SAM suite to check the requested software is installed, Vladimir reported problems
with crashing jobs so Stuart agreed to investigate.
- Action(151) - Update of central BK components with optimized methods
Ongoing.
- Action(156) - Refine VO-box setup to minimize impact of updates and maintenance procedures
Raja will test his script against v4r17p1. Action closed.
- Action(161) - Define Production Request for ongoing production to allow its monitoring
Action closed.
- Action(162) - Document the TQ priorities algorithm
Ongoing.
- Action(163) - Investigate possible use of CERN backup tools for log files archiving.
Ongoing.
- Action(164) - Web portal release procedure
Ongoing.
-Action(165) - Add WMS accounting for visualization of 'time to running' and other accounting metrics
Ongoing, Ricardo is working on it.
-Action(166) - Set up volhcb05 as the new certification machine
RPMs were installed on volhcb05 (SLC4) but this caused inconsistencies. Ricardo would like to reinstall that machine
from scratch in order to proceed, Roberto will contact FIO to arrange this.
-Action(167) - Ensure new middleware bindings are available and start SLC5 commissioning on volhcb12
The bindings are present but still not working correctly. Stuart and Vladimir (were) volunteered to run small tests to
run on the 5 SLC5 sites (with SLC5 native and SLC4 in compatibility mode). There was some discussion regarding
voms-proxy-init and whether this works at all sites.
2. Production Status Report
- Report from the Production meeting
Still waiting for disk space (and removal of backlog of requests) before restarting. There are several new requests
pending and others that look to have produced enough events should be validated before proceeding. Production
validation should be added as a phase in the production cycle and an agent instrumented to check and resolve any
inconsistencies.
- Data destination SE definition
Stuart reported that this was already resolved, new simulation productions will be created with the policy of
attempting any T1 SE in the associated grouping before going to failover.
- v4r18 release
Andrei mentioned some production DB optimizations and other fixes are part of v4r18 which will be imminently
released.
- Production priorities discussion
See discussion triggered on the lhcb-dirac mailing list. Andrei agreed to create a twiki with the summary of this
discussion.
3. DIRAC Tasks
Items discussed above are omitted from this summary.
- Infrastructure monitoring (SAM,SLS,Dashboard)
Roberto is producing a description of the SAM suite for site admins and other interested parties. Monitoring and
infrastructure tests should be migrated to Nagios and this is also being investigated. Federico is almost ready to
deploy the first iteration of his suite for gathering & publishing information on sites and resources etc.
- SystemLogging
The system logging database schema was updated, Andrei suggested dropping the existing schema and starting
afresh with the new one. Marcos & Victor agreed this was fine.
- Framework status
Threaded tasks inside threaded tasks were causing a memory leak in the sandbox service and this is still under
investigation. The proxy length issue fix should now be in place.
- Web Portal
Philippe asked the status of the BK web interface and Matvey conveyed that he was unsure how to save files. Andrew
commented that the CS can be saved as a text file.
- WMS status
Andrei asked what was necessary in order to migrate purely to the new sandbox client. It was agreed to check with
the Ganga developers to see if they are using the correct DIRAC client version before proceeding with this. Others at
external sites, for example, would have to update their DIRAC installations. Philippe agreed to set up a distributed
analysis meeting in order to check this. The WMS cleaning agent policy should be amended to not clean production
jobs and a production cleaning agent should be put in place.
- Services deployment ( hardware, configuration )
Discussions with IT people resulted in tests that could be made on volhcb14. A frontend machine was requested to
place load on this DB backend.
4. AOB
Meeting finished at 1pm.
There are minutes attached to this event.
Show them.