Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-014
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 5th May 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: A.Tsaregorodtsev
CERN: Z.Mathe,R.Santinelli,F.Stagni,J.Closier,A.C.Smith,Ph.Charpentier, R.Nanadakumar, R.Vazquez
Phone: R.Graciani, A.Tsaregorodtsev
0. Ongoing Action List
(94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - steer the CPU related activities in DIRAC, document procedures, move to the HEP CPU units
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(132) - 03/03/09 - Stuart - Update LFN construction utility to store production output directories as a parameter
(133) - 03/03/09 - Matvey - Provide request DB interface page
(139) - 31/03/09 - Roberto,Stuart - Track and test glexec as it is deployed at the sites, put CE into production pending the official release
(140) - 31/03/09 - Andrew - Formulation of procedure for how to treat banning of sites based on job requirements e.g. input data / staging...
(142) - 14/04/09 - Adri - verify that the data transfer monitoring info sent to SiteMonitoring is functioning
(143) - 14/04/09 - Andrei - update the links in the job monitoring info sent to SiteMonitoring
(144) - 21/04/09 - Zoltan - propose and implement solutions to reduce the Bookkeeping service load for production input data queries
(145) - 21/04/09 - Vladimir,Roberto,Joel - check why SAM application test jobs are running by sgm user on some sites
(146) - 21/04/09 - Vladimir,Joel - SAM SW installation logic by synchronizing shared areas with the CS SW specification
(147) - 21/04/09 - Joel - renew the request to IT for deployment of the new LHCb/DIRAC central host machine
(148) - 21/04/09 - Joel - setup DIRAC SVN/Trac project portal and configure access to the DIRAC developer team.
(149) - 05/05/09 - Ricardo - test the --use-grid SetupProject option to set the LFC_HOST to be used by CORAL
(150) - 05/05/09 - Roberto - update the Production Procedures TWiKi with details on GGUS ticket submission
Actions closed:
(124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side and protect data operations
(126) - 20/01/09 - Andrei - Document the design of the workflow / production management components
(136) - 03/03/09 - Vladimir,Roberto,Joel - Integration of existing SAM infrastructure tests into DIRAC SAM framework.
Check the Nagios based SAM framework deployment time scale.
1. Action Updates
- Action(94) -
Ongoing.
- Action(111) -
Ongoing
- Action(116) - CPU studies
Ongoing. Ricardo reported that the CPUTimeLeft utility is updated to return the results in the same units
as used in the pilot JDL for the MaxCPU time specification ( 500 SI2K CPU equivalent ). THis can be used for
the Filling mode, but the mode is by default switched off before more tests to be done. The watchdog of the user
application is instrumented to send KILL signal to the application shortly ( 2 cycles ) before the CPU
limit is reached as reported by the batch system. Next steps:
- express all the timing parameters in the HEP Spec units;
- report the CPU time consumed in normalized units based both on the Site configuration normalization factors
and on the LHCb benchmarking.
- Action(122) -
Ongoing. Ricardo provided Vladimit with a recipe how to obtain the pilot references from the Logging service
information
- Action(124) - Apply dataset limit check
The limit is applied on the services side, failing jobs if necessary. Point closed.
- Action(126) -
Ongoing activity to be discussed in the DIRAC tasks. The point is closed as an action.
- Action(132) -
Ongoing with lower priority
- Action(133) - RequestDB monitoring
Ongoing.
- Action(136)
Moved to infrastructure DIRAC task as an ongoing activity.
- Action(139) -
Ongoing, pending more sites deploying glexec. Next candidate for testing - IN2P3
- Action(140) - Banning faulty storage elements
Ongoing.
- Action(141) - done and closed
- Action(142) - data transfer monitoring info sent to SiteMonitoring
Ongoing.
- Action(143) - links in the job monitoring for SiteMonitoring
Ongoing.
- Action(144) - Bookkeeping query optimization for the production management
Ongoing.
- Action(145) - Check user roles for the SAM App test jobs at sites
Ongoing.
- Action(146) - SW installation framework using CS data
Ongoing. The task should complemented with the per site SW monitoring tools.
- Action(147) - new DIRAC service host
Joel has got access to the new machine and is preparing it for the production
- Action(148) - SVN/Trac server
Ongoing.
2. Production issues
- Production status
DIRAC release v4r13 is installed in production as a necessary prerequisite to start the MC'09 production.
2 production are defined with and without MC truth information. A 10% failure rate is observed.
Some jobs are failing at SLC5 WN's. It was decided that SLC5 CE's will be removed from the site
definitions in the CS (Action: Joel).
LogSE is getting full. Joel will update the automatic cleaning agent to clean both /lhcb/test
and /lhcb/MC/test areas. Recent dummy production areas will be also cleaned.
To lower the load on the LFC master service at CERN, the GaudiApplications should be configured
to use other LFC mirrors by the CORAL layer. This can be done either by setting --use-grid option
with SetupProject or setting LFC_HOST environment variable explicitly. Ricardo will try out the
SetupProject option. This should set the LFC_HOST to point to the CERN readonly LFC instance.
3. DIRAC Tasks
- Infrastructure monitoring (SAM,SLS,Dashboard)
No update
- Framework status
Ricardo reported on the recent developments:
- possibility to abort data transfers on the server side
- improvements in the voms proxy generation with spotting potential problems
- Bookkeeping status
Zoltan reported a bug fix in the client preventing interrogation of the MDF files. This
will be fixed in the v4r13p1 patch release. Bookkeeping clean-up is to be done to remove
faulty FEST configurations.
- WMS status
A quick discussion on the entry check of the jobs with input data resulted in a consensus to
defer the file metadata retrieval to the JobWrapper. This should reduce the time of the input
data check by a factor of ~5.
Reviewing and defining tools to set alarms and notifications in DIRAC is identified as one
of the topics for the upcoming DIRAC week.
Ricardo reported that the possibility to set the path for the output data in the LFC namespace
is added to the DIRAC API.
- DMS
A proposal to allow users to set comments to their files in the LFC was not well accepted.
It was decided to postpone this and rather use the file name space for the user metadata.
- Production Management status
No update
- VO-boxes
VO-boxes should be upgraded to v4r13 ( ACTION Raja ).
- Web Portal
No update
- Services deployment ( hardware, configuration )
No update
- Documentation
No update
4. AOB
There was a discussion on the practice of sending GGUS tickets. It was decided that just Grid Experts
will be allowed to send ALARM tickets. Shifters and developers can all send TEAM tickets. Tickets
related to T0 and T1 sites should have lhcb-grid mailing list in CC to allow Grid Experts to follow
them. Tickets to T2 sites should be followed by the submitters. The TWiKi Production procedures page
will be updated with this amendment.
Joel reported that the GGUS portal will allow access based on the VOMS roles rather than groups.
This will make it a lot easier for us since we are not using groups.
It was reminded to all the CHEP paper authors that the paper submission deadline is at 15 May.
The papers should be prepared first as LHCb notes and approved by the Project Leader to follow
standard procedures. They should be also uploaded to the CHEP Indico page.
There are minutes attached to this event.
Show them.