Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 1st April 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, J.Closier, R.Graciani, G.Kuznetszov, E.Lanciotti, S.Paterson
Phone: R.Bernet, N.Brook, U.Marconi, M.A.S.Miguelez, M.Sapunov, A.Tsaregorodtsev
Apologies: P.Charpentier, R.Santinelli
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne, Joel - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(46) - 11/03/08 - Angelo, Roberto - Send some test jobs to CNAF to test StoRM
(48) - 18/03/08 - Roberto, Gianluca - Investigate the reason for multiple pilots aborted after successful "list-match" check
(49) - 18/03/08 - Roberto - Develop and deploy a sensor checking the SAM DB site info against lcg-infosites
(50) - 01/04/08 - Gianluca - Investigate the software installation / availability failures of recent production jobs
(51) - 01/04/08 - Joel - Prepare production tool 'wish-list' in advance of DIRAC3 week
(52) - 01/04/08 - Stuart, Andrei - Plan Production Manager training session
(53) - 01/04/08 - Stuart - Introduce splitting of large bulk LFC queries in DIRAC API
(00) - 01/04/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (41) - 26/02/08 - Marianne - Trigger replication of stripped data to other Tier-1s (RAL, IN2P3)
Done - (44) - 04/03/08 - Roberto - Contact NIKHEF about increasing the length of available queues
Done - (47) - 18/03/08 - Andrei - Update the proxy renewal procedure on the analysis system (DIRAC v2r19)
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Gianluca reported progress to the Operations eLog, 40K files in the ProcDB seem to be ok. 2K files statuses failed to
update with the CLI for an unknown reason. Andrei suggested further investigating the cause of this failure.
Action(01) now merged with Action(42).
- RAL / PIC migrations - Action(04)
Marianne mentioned that for RAL, Raja was asked to remove two tapes that failed in the past and these FTS jobs
have been resubmitted. There are still 46 tapes remaining to be migrated at a rate of 2 per day. Around 2K DIGI
files at PIC are still being recovered, the timescale for this is ~1week.
- Bookkeeping prototype - Action(07)
Elisa has started to migrate the DC06 data with a new method that greatly improves the speed (~40 rows / second).
With a subset of the ~28M rows requiring migration the timescale is difficult to determine. Andrei asked how long it
could take before a full-scale test of the prototype could take place and Elisa conveyed that the CCRC phase 2 in May
may be too early for the full functionality. Population of the prototype (without creation of views etc.) could still take
place. Action(07) Now Merged with Action(09)
- Monitoring framework - Action(09)
Roberto has compiled a detailed list of WLCG services used in Grid including their criticality for LHCb and is awaiting
feedback from our side. The next step is to work with Julia's team to provide a prototype based on Service Map.
With Andrea Sciaba Pablo Saiz and Alessandro Di Girilamo (ATLAS) there is now a small team for gathering all
experiment's requirements about a Service Map. Ricardo mentioned that the Service Map should be complemented
with DIRAC services. Action(09) now merged with Action(36).
- Test jobs to CNAF to test StoRM - Action(46)
Pending the end of the CNAF downtime.
- Update the proxy renewal procedure on the analysis system (DIRAC v2r19) - Action(47)
The proxy renewal procedure was updated by Andrei.
- Investigate Aborted Pilots - Action(48)
Debugging on the client side has not shed any light on the cause of this problem. This should be pursued on the
server side if possible.
- Develop sensor checking SAM DB against lcg-infosites - Action(49)
Roberto triggered SAM support to look at the list of CEs supposed to support LHCb and appearing in SAM but not in
the BDII.
- Zero event files
Marianne has been investigating the cause of the discrepancy between the number of events processed and reported
in the Bookkeeping. The files have been processed successfully and Marianne is continuing the cleanup. Another
case was found where stripped data was missing from the BK since the ancestor files were missing. Simply adding
the ancestor file allows the stripped data to be entered.
Gianluca discovered 400 jobs recently completed with Post-Mortem Finalization due to an application error. Roland
confirmed that this is related to the software installation and it was agreed to check with the affected sites that the
shared file systems are powerful enough or whether this is related to the software installation. ( ACTION: Gianluca )
The problems with stalled jobs at IN2P3 appear to persist with several stalled jobs in evidence. Andrei is continuing
to debug with local access at the site.
There was some discussion about the training of the production management team prior to the second CCRC phase.
The tools for the production manager must be defined and Joel agreed to prepare a wish list. ( ACTION: Joel ) It was
also agreed to discuss the preparation of a production manager training session, the Wednesday morning of LHCb
week was tentatively suggested. ( ACTION: Stuart, Andrei )
2. Production Operations Issues
- Using LFC with bulk queries from Ganga
Job splitting on the client side with several thousand files can take O(hours) which is too long for the users. Andrei
shortened the timeout between queries in the LFC combined client that should improve the performance. Splitting of
bulk requests into e.g. 500 file chunks should be added to the DIRAC3 API and back-ported to DIRAC2. ( ACTION:
Stuart )
3. CCRC2 Schedule
The CCRC F2F presentation from Nick is attached to the agenda page.
4. DIRAC3 Tasks
- Stripping tests preparation
Joel reported that no new progress has been made since last week with the stripping workflow.
- AncestorFiles & CondDB Optimizers
Stuart reported that CondDB and AncestorFiles optimizers were put in place last Friday. The CondDB optimizer
checks available tags in the LFC and holds jobs for a predefined time period before failing them if not available. The
AncestorFiles optimizer employs the old genCatalog utility to query the BK for ancestors.
- Stager status
Stuart conveyed some minor schema changes have been made to the StagerDB but the Stager service is now running
and the Agents to make staging requests and montior the progress are being developed.
- Copying input data locally to WN
Data access issues at several sites have prompted the inclusion of a module to download data local to the WN. This
can be a site specific policy of LHCb.
- Failover status
The workflows should be complemented with failover of data uploads and other operations such as status transitions.
Andrei has spent some time investigating workflows as a solution to generic requests. This is currently being
finalized.
- Accounting, Distribution and Installation tools status
The status of these tools will be confirmed during the developer week. The installation procedure should be simplified
as much as possible for VO-boxes and clients.
5. AOB
Andrei reminded the developers that the DIRAC3 week is scheduled for 7th April.
There are minutes attached to this event.
Show them.