Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 15th January 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, U.Marconi, S.Paterson, R.Santinelli, A.C.Smith
Phone: R.Bernet, N.Brook, A.Casajus, G.Kuznetszov, M.A.S.Miguelez, M.Sapunov, R.Nandakumar, A.Tsaregorodtsev
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne - Remedy the Exceeded Max Staging Retry jobs and update ProcDB (pending resolution of each case)
(02) - 06/11/07 - Gianluca, Marianne - Agents for automatic resolution of IntegrityDB entries should be developed
(03) - 30/10/07 - Marianne - Update of LFC replica flags (pending NIKHEF)
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Elisa, Marianne - Prototype bookkeeping with API wrapped in DISET service after ensuring AMGA and Oracle availability on volhcb07
(08) - 04/12/07 - Gianluca, Alexei, Andrei - Checking sporadic failures at GridKa are now pending tests with latest DIRAC version
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop LHCb SAM framework and summarize architecture for SE tests
(21) - 18/12/07 - Roberto, Gianluca - Test RFIO at CNAF, coordinate with local site contacts
(23) - 08/01/08 - Nick, Marianne, Raja - Arrange a meeting with RAL management regarding the migration activity
(25) - 08/01/08 - Marianne - Resolve why number of processed events is missing in Bookkeeping (also cause of missing files)
(26) - 08/01/08 - Joel - Oversee upgrade of volhcb01 now that BK and LogSE are backed up
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(28) - 15/01/08 - Nick - Trigger arrangement of meeting between RAL, CNAF and Castor team (see Action (21))
(29) - 15/01/08 - Gianluca - Remove lost files at PIC from ProcDB
(30) - 15/01/08 - Gianluca - Test NIKHEF after reinstallation of the shared area
(00) - 15/01/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (06) - 27/11/07 - Andrei - Prepare draft pilot document as internal LHCb note
Done - (11) - 11/12/07 - Marianne - Clean up SIM/DIGI for production IDs 1957, 1960, 1962
Done - (14) - 11/12/07 - Joel - Update lhcb-proxy-init with small wrapper to ensure environment is not changed
Done - (17) - 11/12/07 - Joel - Prepare MDF workflows (test with v2r18p4 / v2r18p5)
Done - (18) - 18/12/07 - Gianluca - Check current status of SAM jobs on Test system
Done - (19) - 18/12/07 - Roberto,Joel - Prepare draft of the LFC Admin service interface, comments welcome via email
Done - (24) - 08/01/08 - Philippe,Roberto - Investigate setting up an AMGA server on the PPS
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Work is ongoing, the update of the ProcDB is now urgent and Andrei suggested Gianluca should estimate the effort to
resolve this.
- Agents for automatic resolution of IntegrityDB entries - Action(02)
There is a clear solution for some cases but not all, this work is ongoing while all cases are resolved.
- MDF wokflows - Action(17)
Joel launched the production yesterday evening and has conveyed the files to Radu to begin transfers. CERN and
IN2P3 are capable of running the jobs. PIC, CNAF and RAL are banned from the ProcDB to avoid problems with
migrations etc.
The list of files lost at PIC was supplied to Gianluca by Marianne. These files are currently being removed from the
ProcessingDB. ( ACTION: Gianluca ) A problem with the proxy renewal was reported to Matvey and Andrei. Rather
than back-porting code to fix this from DIRAC3 it was agreed to use long proxies in DIRAC2 for now. The corruption of
the shared area at NIKHEF was solved by reinstallation via the SAM tests. This should be tested by rescheduling failed
jobs. ( ACTION: Gianluca ) The LFC intervention has been postponed and is now foreseen for Thursday. Operations
and CCRC08 logbooks were suggested by Philippe and several proposals have been made. It was agreed to
standardize on the 'elog' (also used by the Online and detector groups).
2. Production Operations Issues
- RAL and PIC Migrations - Action(04) & Action(23)
Marianne mentioned that the RAL-disk migration is ongoing and problematic files left over from transfers are being
resolved. All files should be checked before releasing the pool. The current estimate is that next week the disk
migration will be complete. For RAL-tape, the site admins have been staging the files manually but these are not
coming online due to a Castor problem. Nick contacted Andrew Sampson and this was pending until the progress of
tape staging was checked. A summary of all the issues to discuss at this meeting should be compiled by Marianne and
the agenda should be compiled by Nick ( ACTION: Marianne, Raja, Nick) Roberto checked with Gonzalo and the PIC
migration is now scheduled for the week of 21st January.
- LFC mirrors - Action(05)
Roberto left his test suite running over the break and observed 100% efficiency over the Tier-1 LFC instances
corresponding to a constant 1Hz rate. The replication of user and group info tables was also successfully tested. The
IN2P3 instance is now in production and NIKHEF / PIC mirrors are planned for the middle of March after the first CCRC
phase. Alexey is following the plans at GridKa. The remaining LFC instances can now be added to the production
mask. This action will be put on hold until after the first CCRC phase.
- Pilot document - Action(06)
The draft is attached to the agenda for today.
- BK prototype - Action(07) & Action(24)
Progressing with port-opening and installation of Bookkeeping. Joel will contact the IT people to trigger installation of
Apache on volhcb07. Setting up an AMGA server on the PPS will take the order of months rather than weeks so the
current instance will be used. For prototyping purposes, Oracle can be temporarily dropped if this is a stumbling
block. Andrei suggested that Marianne should report on the preparatory work for the meeting next week.
- Sporadic failures at GridKa - Action(08)
Joel has performed preliminary tests last Friday that were successful. This should be rechecked with the latest version
of Brunel. ( ACTION: Gianluca )
- RFIO at CNAF - Action(21)
Roberto reported the problem with the highest possible priority to the WLCG Operations meeting and also to the
CCRC08 meeting since this has rfio at CNAF has not been functional since September 2007. It was suggested to
trigger the arrangement of a meeting with the CNAF and RAL site admins as well as the Castor team such that a
solution can be found without duplicating effort. ( ACTION: Nick )
3. DIRAC3
- Distribution of externals
Joel mentioned that work is ongoing for the SRM2 external packages. Bundling a UI independent self-consistent set of
libraries is of the upmost priority for DIRAC3.
- WMS job execution chain status
Stuart has written an AgentMonitor that uses the ThreadPool to keep track of pilots. Reorganisation of the input data
resolution in the Job Wrapper and other Agent side improvements are pending the stable distribution of externals.
- Workflow and Production Management status
Gennady can now submit jobs but using workflow as a persistent object is to be discussed with Stuart. Job
submission through the API is working. Matvey has started to construct the production monitoring page and Joel /
Gennady are asked to take a look at this and make any comments.
- Recent CS problems
There have been problems with intermittent failures during all components of the framework recently. Adri wil
investigate with Andrei and Stuart this afternoon.
- Data management status
Andrew commented that components from DIRAC2 are being ported to DIRAC3. The RequestDB required a few
modifications and the TransferAgent has to be updated to the new ReplicaManager. This should hopefully be
completed by the end of the week. The upgrade of the RequestDB instances on the VO-boxes should be performed
(starting with Lyon for debugging and can be reserved for DIRAC3 use). Adri will investigate opening of a further port
on the PIC VO-box.
- Services monitoring status (plots, logging service)
Adri is half-way through completing a document about the service monitoring. This will be circulated when available.
Raja, Marcos and Matvey are encouraged to propose a consistent picture of service monitoring.
4. AOB
None
There are minutes attached to this event.
Show them.