Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 20th May 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Adinolfi, M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, E.Lanciotti, Z.Mathe, S.Paterson, R.Santinelli, A.C.Smith
Phone: R.Bernet, N.Brook, R.Graciani, R.Nandakumar, A.Tsaregorodtsev
Apologies:
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne, Joel - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne, Andrew - RAL, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4, port IntegrityDB to DIRAC3
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(56) - 29/04/08 - Roland, Stuart, Joel - Provide Production Manager guide
(58) - 13/05/08 - Joel - Resolve issue with shared area at LCG.Brunel.uk
(59) - 13/05/08 - Andrei, Angelo - Check StoRM SE definition in DIRAC2
(60) - 13/05/08 - Andrei - Add LFC synchronization to addDirectory call
(61) - 20/05/08 - Roberto - Investigate dashboard SAM visualization tool
(00) - 20/05/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (57) - 13/05/08 - Roberto - Check the DIRAC3 service status reporting on the GridMap page
Done - (59) - 13/05/08 - Andrei, Angelo - Check StoRM SE definition in DIRAC2
1. CCRC Status
Nick agreed to take minutes for this session and post them to the LHCb CCRC'08 ELOG.
2. Production Status Report
Minutes from the production meeting are available here.
- ProcDB update - Action(01)
Philippe suggested continuously running the agent to check the jobs for which files could not be staged. Files in an
assigned state seem to be a problem, for the L0 stripping there were 20K files assigned but no jobs running. Andrew
raised the point that many files for the L0 stripping have been entered in the LFC but the requests are pending in the
Bookkeeping. Philippe is recreating the jobs from assigned files and is up to 300 jobs for the L0. Due to the bug in
the CheckLogFile, files were inadvertently marked as processed instead of problematic. Philippe has recovered a few
files in this state and marked them as unused. A new CheckLogFile has been provided but not yet tested.
- RAL Migration - Action(04)
Andrew indicated that only 20 tapes are remaining for the RAL migration. The major problem over the past 6 months
is that the dCache garbage collector was removing files from the cache that were still to be transferred, while
preferentially keeping those that had already been transferred. Removing the files from the cache that had been
transferred by hand (via an SRM2 door) the progress was improved. Raja commented that a problem with the RAL
tape drive resulted in a reboot yesterday morning. As discussed in the CCRC'08 meeting, the load on volhcb03 has
been a limiting factor and there are plans to migrate the DMS services to another machine. There was some
discussion about the pending transfers on the VO-boxes. Raja reported many sites are banned in the SE mask and
this is blocking all progress. Philippe mentioned that several requests were failing for the past several months due to
a bug in the treatment of larger file sizes.
- Bookkeeping prototype - Action(07)
Elisa has finished the migration of the data to the new schema is completed. Philippe highlighted that when there are
problems entering given information such as the productionID, these files are not visible in the views. A decision has
to be made on how to proceed with this in the new schema. After some discussion it was decided to investigate the
problem at the level of the BookkeepingReport module. Elisa was asked to proceed with correcting these entries.
Zoltan is preparing tests to check the AMGA performance.
- Monitoring framework - Action(09) and Action(57)
Roberto registered several new sensors in SLS. The comments from Andrei and Philippe regarding the Service Map
will be implemented. There was a request to investigate the Dashboard SAM visualization tool. ( ACTION: Roberto )
- Backup DMS services on lxgate34, port IntegrityDB to DIRAC3 - Action(27)
Andrew will attempt to make a start on this as soon as possible. Gianluca has not made any recent progress in
developing the tools for the resolution of the IntegrityDB. Philippe asked whether any entries are cleaned and what
actions are performed with respect to the catalogs. Gianluca confirmed that no files were removed from the DB or
catalogs. Andrei suggested that Gianluca's script should be run in a mode where problematic replicas are
re-replicated and successfully resolved replicas are removed from the DB.
- Recover data that failed to be registered in the Bookkeeping - Action(55)
Marianne reported a large number of failed requests were present in the old Bookkeeping infrastructure. Some of
these have been added by hand but work is ongoing. The DC04 cleanup is underway and the RTTC cleanup is
pending.
- Add LFC synchronization to addDirectory call - Action(60)
Philippe will confirm the status of this after testing the fixes put in place by Andrei.
3. Production Operations Issues
- CNAF StoRM access for analysis
Stuart suggested checking the version of DIRAC2 on the User system against the Test system to solve the problem
with lcg-gt for file protocol.
- DC06 Productions Progress
Philippe conveyed that the resolution of file statuses in the ProcDB resulted in several jobs successfully running last
week. There was some discussion on the workflow strategy with regard to reporting to the ProcDB and BK reports,
this should be performed at the very end of the chain and must carefully check the status of all the prior steps.
Philippe is also investigating why some reconstruction productions were over-produced.
4. DIRAC3 Tasks
See CCRC'08 minutes.
5. AOB
There was a suggestion to begin the PASTE meeting at a later time since the 9:30am start has only prolonged the
meetings. Andrei decided to keep the 9:30am start and make every attempt to enforce a finishing time of 11:00am.
There are minutes attached to this event.
Show them.