Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 12th June 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, S.Paterson, N.Pukhaeva, R.Santinelli, A.Smith
Phone: N.Brook, R.Graciani, U.Marconi, M.A.S.Miguelez, R.Nandakumar, M.Sapunov, A.Tsaregorodtsev
1. Production Status Report
Minutes from the production meeting are available here.
Joel has resubmitted the stripping jobs with the same CPU requirement as simulation. This will allow pilots submitted
for simulation jobs to pick up higher priority stripping jobs at suitable sites. A summary by site is as follows:
- CNAF: Castor is back but no jobs can run. The staging has been very efficient but only a small number of CPU was
published so no slots available. Joel has sent an official email and agreed to follow this up. ( ACTION: Joel )
- CERN / PIC: there has been steady but slow progress for CERN and PIC. At CERN it looks as if tapes were blocked,
Joel commented that the next Castor intervention will resolve this issue.
- NIKHEF: mapping problems are still being observed, the investigations are ongoing. ( ACTION: Joel )
The Bookkeeping views have not been recently updated. ( ACTION: Carmine ) Ricardo mentioned that several jobs at
CERN had failed with post-mortem finalization, Stuart confirmed that v2r15 (including the root protocol fix) had not
been put into production yet pending Joel's test jobs which were in the system.
In order to proceed with using production manager credentials it was agreed to develop a test suite for all the Tier-1
sites such that any inconsistencies can be checked before production managers submit jobs. The recent problems with
YAIM affecting LHCb mappings are to be investigated. ( ACTION: Roberto ) A new release is expected next week which
will allow VO control of mappings in the CIC portal. Andrei asked how this was deployed at the sites and Roberto
mentioned that site admins currently run a script by hand for the updates.
Several sites in the UK (Manchester, QMUL, ...) have recently been failing to install software. Many sites also had a
missing Gauss version (tags were not published). The shared areas at these sites either were not writable or have
disappeared. Joel will continue to debug these problems. ( ACTION: Joel )
2. Production Operations Issues
The v2r15 distribution is ready but not yet tested, Joel has sent test jobs on prestaged data to confirm this is ok. The
v2r15 client should also be released on lxplus. ( ACTION: Joel )
Marianne reported on the DC06 cleanup status. DST distribution to RAL, IN2P3 and NIKHEF is complete and PIC /
CNAF will be finished by the end of the day. As yet there is no news about GridKa and Roberto agreed to escalate this
issue. Andrew suggested contacting Sven Herman (ROC manager) and Marianne agreed to contact him. ( ACTION:
Marianne ) Umberto and Philippe also suggested notifying Michael Schmelling.
Joel agreed to look at a suitable SAM test to check software tag publication. Roberto mentioned that no serious
objections were made to adding srm-get-metadata to the ops critical tests. Historical plots will also be included as part
of the check.
Roberto raised the question of why we are not currently using the gLite RB. After some discussion it became clear
that other experiment VOs do not use the CERN instances of the RBs and several patches are required for LHCb to use
these. The CNAF gLite RB is the latest version, however, this is not deployed everywhere. Stuart suggested LHCb
should wait for consistent versions to be everywhere before switching to the gLite RB. This was agreed and Gianluca
can continue any testing in parallel. Nick mentioned that having RB versions deployed on the PPS is not useful for
LHCb. We must set up and maintain separate machinery to test on the PPS. Imperial are yet to complete the testing
of the gLite RB and Nick agreed to raise the issue at the next GridPP MB. ( ACTION: Nick )
Ricardo mentioned that the gLite Agent Director module was consuming a very large amount of CPU on the
production system. This is because there was no limit in place on the number of jobs selected per loop from the
JobDB, Gianluca agreed to resolve this. ( ACTION: Gianluca )
A flaw in the proxy renewal mechanism was discovered. Many jobs in the User and Test systems are failing due to
expired proxies (Production is ok since Raja's credentials are currently in use). Stuart questioned whether the script
to renew VOMS proxies needed to be in Perl and it was agreed that this should be written in Python and incorporated
in the LCG utilities. Andrei mentioned that this should be included in the v2r15 release and pursued with the utmost
priority. ( ACTION: Gianluca )
During the computing meeting in the last LHCb week, Joel presented some basic rules for accessing Grid resources.
Andrei asked for one or two paragraphs to be added to the Grid AUP clarifying the LHCb specific policies. Nick
mentioned that publication of user Grid usage should be part of this. Joel agreed to clarify with Philippe who should
prepare a draft of this. ( ACTION: Joel )
A dataset proposal was attached to the Agenda by Andrew. After some discussion it was agreed to amend the
proposal as necessary and further comments are welcome via email to the lists. ( ACTION: Andrew ) A summary of
the currently available File Catalogue client methods was requested from Marianne in order to establish the required
changes for datasets. ( ACTION: Marianne )
3. DIRAC3 Tasks
Ricardo mentioned that several patches to the framework have been deployed and the web-pages are underway.
Work on the logging service is expected to start this week.
Andrei reported that jobs have been submitted through the DIRAC3 job receiver but some problems had been
observed with sandboxes. This is ongoing.
Stuart proposed the creation of the JobSanityAgent (initially in DIRAC2 but also for DIRAC3) in order to resolve issues
such as the following:
- Checking for output data already existing in the LFC
- Failing jobs with problematic JDL (not existing or illegal)
- Checking that input sandboxes are correctly uploaded (non-zero size)
- Failing jobs with too much input data specified (e.g. jobs with > 200 files)
- Failing jobs with input data not specified according to LHCb conventions (e.g. not /lhcb/)
( ACTION: Stuart )
Andrew raised the issue of using the Stager for cache management and it was agreed to formulate a proposal in the
coming weeks about this. ( ACTION: Andrew, Stuart )
There was a feature request for the current Stager to reflect progress in a simple way. Stuart said that this would be
added to the StagerAgent and deployed as quickly as possible using the MinorStatus field in the monitoring. (
ACTION: Stuart )
4. AOB
Philippe conveyed that the CB accepted the change of project leadership. Thomas Ruf has been appointed as Physics
Software Project Leader and Philippe as Core Computing and Software (CCS) Project Leader. On behalf of everyone
present, Philippe thanked Nick for the past two and a half years leadership and also thanked Andrei for chairing the
NCB. Umberto will now take over as NCB chair.
There are minutes attached to this event.
Show them.