Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 22nd January 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, A.Casajus, G.Castellani, P.Charpentier, J.Closier, G.Kuznetszov, U.Marconi, S.Paterson, R.Santinelli, M.Sapunov, A.C.Smith, A.Tsaregorodtsev
Phone: R.Bernet, N.Brook
0. Ongoing Action List
(01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this (merged with Action(02))
(03) - 30/10/07 - Marianne - Update of LFC replica flags (pending NIKHEF)
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Elisa, Marianne - Prototype bookkeeping with API wrapped in DISET service after ensuring AMGA and Oracle availability on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop LHCb SAM framework and summarize architecture for SE tests
(25) - 08/01/08 - Marianne - Resolve why number of processed events is missing in Bookkeeping (also cause of missing files)
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(29) - 15/01/08 - Gianluca - Remove lost files at PIC from ProcDB
(30) - 15/01/08 - Gianluca - Test NIKHEF after reinstallation of the shared area
(31) - 22/01/08 - Roberto - Investigate cause of CERN RB NS behaviour
(32) - 22/01/08 - Gianluca, Marianne - Identify and resolve faulty productions
(00) - 22/01/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (08) - 04/12/07 - Joel, Alexei, Andrei - Checking sporadic failures at GridKa are now pending tests with latest DIRAC version
Done - (21) - 18/12/07 - Roberto, Gianluca - Test RFIO at CNAF, coordinate with local site contacts
Done - (23) - 08/01/08 - Nick, Marianne, Raja - Arrange a meeting with RAL management regarding the migration activity
Done - (26) - 08/01/08 - Joel - Oversee upgrade of volhcb01 now that BK and LogSE are backed up
Done - (28) - 15/01/08 - Nick - Trigger arrangement of meeting between RAL, CNAF and Castor team (see Action (21))
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01) and (02)
The Agent logic has to be improved after the recent corruption of the Processing DB.
- GridKa - Action(08)
Joel confirmed that recent tests have been successful.
- Event number discrepancy in BK - Action(25)
The number of events reported by the BK is not displayed accurately by the BK Web Interface. This should be
identified and resolved first on the prototype machine before deploying the fix to the production instance.
- NIKHEF - Action(30)
Due to the recent RB NS problems, the testing is ongoing. Roberto will investigate why it seems that the NS is
restarted frequently and how to react in this case. ( ACTION: Roberto )
Joel summarized that all sites except NIKHEF are now running stripping activities. The problem of running at PIC was
resolved with Adri triggering a recursive permissions change on the shared area. Hubert has updated the software
distribution tarball.
2. Production Operations Issues
- RAL and PIC Migrations - Action(04) & Action(23)
Andrew noted that the RAL-tape FTS jobs were failing over the weekend, the problem seems to be related to
obtaining a GridFTP TURL from the SRM. Raja has been informed about these problems and should track this activity.
Progress has been made since 4am this morning but this is slow. The next steps are to test an XML file for CCRC
testing multiple inputs from the pit. Nick mentioned that since there has been a lack of progress, the meeting today
should be utilized to coordinate activities with the RAL site contacts.
- BK prototype - Action(07)
Marianne has installed the BK on volhcb07 and has started to recreate the views and populate this with a small
amount of data for prototyping. The AMGA service is running in production in Glasgow with a front-end Oracle set up
at CERN. AMGA will be installed on volhcb05 coordinating with Elisa and Birger.
- RFIO at CNAF - Action(21)
Sebastian and Vincenzo remedied the configuration of several servers at CNAF and this has been resolved. Jobs
starting since then are running happily. Philippe conveyed his thanks to Sebastian and Vincenzo for taking time to
resolve this without the help of the local site admins. Philippe also mentioned that only 30 LSF slots are configured
at CNAF per server (CERN has 270 and RAL has 200).
There is a possibility to recover stripped files after the recent corruption of the ProcessingDB leading to duplicated
events. The non-duplicated files should be flagged as unused in this case such that jobs can be reprocessed.
Marianne confirms that removing a given production (Bookkeeping, LFC, etc. ) would not be difficult. Using Philippe's
script it should be possible to identify each problematic production and this could be rerun. Special cases where
1/O(100) jobs are faulty should be corrected by hand. A summary of the operational actions required should be sent
to the production list and the Twiki page updated. ( ACTION: Gianluca, Marianne ) Joel agreed to resubmit the
productions when the cleanup phase is completed.
3. DIRAC3
This week is an ongoing DIRAC3 developers session therefore discussion was postponed in the PASTE meeting.
4. AOB
The next meeting will be the CCRC08 Thursday discussion. Nick raised the issue of the CMS proposal to use the IP
address with SRM2 for dCache. After some discussion it was decided that if dCache is ignoring the space tokens,
action should not be taken against the subsequent disk-to-disk transfer operations on our side. Philippe will raise the
issue at the MB today.
There are minutes attached to this event.
Show them.