Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 8th January 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, P.Charpentier, J.Closier, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, G.Kuznetszov, U.Marconi, M.Sapunov, A.Tsaregorodtsev
Apologies: G.Castellani, A.C.Smith
0. Ongoing Action List
(01) - 06/11/07 - Gianluca, Marianne - Remedy the Exceeded Max Staging Retry jobs and update ProcDB (pending resolution of each case)
(02) - 06/11/07 - Gianluca, Marianne - Agents for automatic resolution of IntegrityDB entries should be developed
(03) - 30/10/07 - Marianne - Update of LFC replica flags (pending NIKHEF)
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(06) - 27/11/07 - Andrei - Prepare draft pilot document as internal LHCb note
(07) - 27/11/07 - Elisa, Marianne - Prototype bookkeeping with API wrapped in DISET service after ensuring AMGA and Oracle availability on volhcb07
(08) - 04/12/07 - Alexei, Andrei - Further investigate cause of sporadic failures at GridKa for production jobs (pending tests with v2r18p5)
(09) - 04/12/07 - Gianluca, Roberto - Prepare an SRM sensor test for the LHCb SAM framework (pending upload to SAM DB)
(11) - 11/12/07 - Marianne - Clean up SIM/DIGI for production IDs 1957, 1960, 1962
(14) - 11/12/07 - Joel - Update lhcb-proxy-init with small wrapper to ensure environment is not changed
(17) - 11/12/07 - Joel - Prepare MDF workflows (test with v2r18p4 / v2r18p5)
(18) - 18/12/07 - Gianluca - Check current status of SAM jobs on Test system
(19) - 18/12/07 - Roberto,Joel - Prepare draft of the LFC Admin service interface
(21) - 18/12/07 - Roberto, Gianluca - Test RFIO at CNAF, coordinate with local site contacts
(23) - 08/01/08 - Nick - Trigger a meeting with RAL management regarding the migration activity
(24) - 08/01/08 - Philippe - Investigate setting up an AMGA server on the PPS
(25) - 08/01/08 - Marianne - Resolve why number of processed events is missing in Bookkeeping
(26) - 08/01/08 - Joel, Marianne - Oversee upgrade of volhcb01 after Bookkeeping is backed-up
(27) - 08/01/08 - Andrew - Check feasibility of installing DMS services on SLC4 to allow migration of lxgate34
(00) - 08/01/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (05) - 27/11/07 - Roberto - Oversee LFC intervention plan for upgrade to latest version (testing)
Done - (16) - 11/12/07 - Joel - Trigger machine upgrades (volhcb06, port 80 not yet visible outside CERN)
Done - (20) - 18/12/07 - Roberto, Andrei - Investigate recent failures of uploading logs (volhcb06) check VO-boxes for Transfer Agent problem
Done - (22) - 18/12/07 - Marcos - 3 new requested productions should be created and submitted, 1911 should be extended
1. Production Status Report
Minutes from the production meeting are available here.
Philippe mentioned a problem at RAL using Castor to read data. When a mixture of Castor and dCache replicas exist in
Stripping jobs at RAL it appears that the Castor files are skipped. It was observed that the files are marked as
'processed' incorrectly in the Processing DB and the Bookkeeping includes these files as input for the job.
Joel raised the issue of the Bookkeeping information presented on the webpage for stripping jobs. The number of
events processed seems to be missing but this has not changed in the Bookkeeping module. ( ACTION 25: Marianne )
- Signal processing and MDF workflows - Action(17)
Joel conveyed that the signal processing workflow is complete and ran over the Christmas break. The MDF workflows
were tested and inconsistent behaviour was observed at GridKa and PIC, Joel will repeat these tests.
2. Production Operations Issues
- RAL and PIC Migrations - Action(04)
Marianne mentioned that the RAL-disk migration is ongoing and problematic files left over from transfers are being
resolved. All files should be checked before releasing the pool. Marianne estimates this would be resolved by the end
of the week. For the RAL-tape migration, Marianne is waiting for an update from Raja / Catalin since the RAL-tape
expert is not yet back from vacations. . Nick suggested involving Andrew Sampson in the discussions for the
migration and this was agreed. ( ACTION 23: Nick ) For the PIC migration everything is ready on our side for the
changes, Roberto checked with Gonzalo and the migration is scheduled for the week of 14th January.
- LFC mirrors - Action(05)
Roberto left his test suite running over the break and observed 100% efficiency over the Tier-1 LFC instances
corresponding to a constant 1Hz rate. The replication of user and group info tables was also successfully tested. The
IN2P3 instance is now in production and NIKHEF / PIC mirrors are planned for the middle of March after the first CCRC
phase. Alexey is following the plans at GridKa. The remaining LFC instances can now be added to the production
mask. This action will be put on hold until after the first CCRC phase.
- Pilot Document - Action(06)
There was no mention of a timescale for the working group on Pilot Agents at the TCG and this will be raised at
tomorrow's meeting by Philippe.
- BK Prototype - Action(07)
An Oracle DB instance and AMGA server are required on volhcb07 to begin testing. Philippe agreed to investigate
setting up an AMGA server on the PPS. ( ACTION 24: Philippe )
- RFIO at CNAF - Action(21)
Roberto commented that a meeting will occur today on how to proceed with rootd installation at CNAF. Philippe
conveyed after discussions with the Castor people that the real problem appears to be a misconfiguration. If root
access at CNAF could be given to the Castor experts, the problem could be more rapidly treated.
- SRM2.2 tests - ongoing and linked with DIRAC3 progress (see below)
- volhcb01 preparation for DIRAC3 production
Joel confirmed that the migration is pending the BK moving to volhcb07. Once done, the port scan and upgrade can
occur. ( ACTION 26: Joel ) volhcb01 will be used in the CCRC08 activity.
- volhcb03 -> volhcb02 migration
volhcb02 will be used for DIRAC3 development. Adria has started to install the DIRAC3 Web Portal on volhcb02.
- lxgate34 status
The migration of lxgate34 is becoming critical and Andrei agreed to contact Andrew about the feasibility of installing
the DIRAC2 DMS services on an SLC4 machine. ( ACTION 27: Andrew )
3. DIRAC3
- WMS Job execution chain status
Stuart commented that the whole chain is working locally but some inconsistent behaviour was being observed on the
Grid so testing continues. The Worfklow library is evolving to contain all VO-specific parts of DIRAC making use of the
plug-in mechanism described at the last PASTE meeting. This will be renamed to the LHCbDIRACLib to reflect this.
- Credentials management status
Andrei has updated and wrapped the tools for credential management in DIRAC3. VOMS conversions and proxy
renewal can now be handled by one module. Andrei confirmed that services can now be secured based on the
ownership and group of the client. This has been put in place for the JobManager service.
- Workflow and Production Management status
Stuart reported that the Gaudi Application module is working fine and complex workflows are possible in the DIRAC3
API. Gennady had started to work on Production Management services before Christmas and made some progress,
the treatment of proxies by this service has led to a small modification in the WMS proxy DB such that Production
Manager credentials can be renewed even when no jobs are present in the system.
- Job Monitoring status
Matvey is working to implement the suggestions from Philippe.
- Services monitoring status (plots, logging service )
Adria circulated a link to the first evolution of service monitoring that could also be extended to report e.g. job activity.
- Data Management status
Basic tools are available and Andrew is currently working on the higher-level services.
Nick queried when DIRAC3 should be declared as ready for CCRC08 and the end of the next DIRAC3 week was agreed
as the deadline.
4. AOB
DIRAC3 developers should now attend the Thursday CCRC08 meetings that replace the Production Operations
meeting.
There are minutes attached to this event.
Show them.