Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 13th November 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, E.Lanciotti, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, G.Kuznetszov, U.Marconi, M.A.S.Miguelez, R.Nandakumar, A.Tsaregorodtsev, A.Zhelezov
Apologies: A.C.Smith
0. Ongoing Action List
23/10/07 - Marianne - Request for basic documentation for restarting the Bookkeeping services
23/10/07 - Marianne - Bookkeeping should be migrated from SLC3 -> SLC4 and installed on volhcb05
23/10/07 - Ricardo - Update Cleaning Agent to prevent user WMS problems
30/10/07 - Marianne - Check number of events for prod 1957 in BK against LFC
30/10/07 - Marianne - Update of LFC replica flags (C checked, N checked but not present)
06/11/07 - Gianluca, Marianne - Remedy the Exceeded Max Staging Retry jobs and update ProcDB as required
06/11/07 - Gianluca, Marianne - Agents for automatic resolution of IntegrityDB entries should be developed
06/11/07 - Raja - Install v2r17p2 on VO-boxes
13/11/07 - Joel - Check dump from Marianne and update ProcessingDB
13/11/07 - Alexei - Produce dump of file namespace at GridKa for Marianne to proceed with LFC replica flags there
13/11/07 - Roberto - Contact site administrators for Tier-1s and install latest version of LFC with Oracle streams enabled
13/11/07 - Roberto - Check status of LFC admin interfaces with WMS instances as trusted hosts
13/11/07 - Roberto - Contact ATLAS representative to understand VOMS usage in other VOs
13/11/07 - Roberto, Gianluca - Resolve issue with Raja's credential causing failed pilot submissions
13/11/07 - Alexei - Investigate recent sporadic failures at GridKa for production jobs
13/11/07 - Joel - To check whether ancestor files are required for reconstruction production 1981
13/11/07 - Roberto - To check the LCG Utilities status for SRM2 at the Tier-1s
13/11/07 - Joel, Matvey - lhcb-proxy-init should be committed and added to the LHCb startup scripts
13/11/07 - Joel - To update VO ID Card to remove obsoleted group definitions
13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
13/11/07 - Andrei - Preparation of the pilot agent framework document
13/11/07 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - 23/10/07 - Marianne - Check dump of ProcessingDB in LFC
Done - 23/10/07 - Andrei - Add increased protection for user job deletion in DIRAC2
Done - 06/11/07 - Stuart - StagingSanityAgent should be developed and deployed
1. Production Status Report
Minutes from the production meeting are available here.
Joel reported that for production 1974, 1975 jobs were taken by Zurich and failed due to a problem with the physics
software. Marco has reported a problem at GridKa with loss of connections to data. Alexei is investigating the cause
of the failed jobs but reports that no failures are observed for NFS access, investigations are ongoing. ( ACTION: Alexei )
Due to VOMS mapping problems, reconstruction was ok for only 3 Tier-1s but since the last meeting, only NIKHEF still
has the issue. The b-inclusive test jobs have been submitted, access to ancestor files seems to be required for these,
Joel will check whether this is necessary. ( ACTION: Joel ) Stripping tests have run successfully so these can start to
be submitted for low lumi. Andrei and Joel will investigate the disappearance of log files for certain jobs. ( ACTION:
Andrei, Joel )
The volhcbYY machines must be rebooted for a kernel upgrade and this will be scheduled for tomorrow morning. This
can start with lxgate03 and volhcb03 to check everything is ok. volhcb02 can now be upgraded to SLC4.
2. Bookkeeping Update
Marianne and Elisa have started to port the Bookkeeping manager service to DISET. Elisa has been trying to
understand how the Bookkeeping works and has started development on the python client for the web page. After
the Bookkeeping migration, volhcb01 can be used as a Test system.
Philippe summarized the proposal from the Bookkeeping working group. The current functionality of the web page
should be replaced with a PyQt interface that can be run in a standalone way. This would allow a better organisation
of queries such as obtaining all files generated by a given version of Brunel.
Andrei asked how to streamline the BK interfaces for the current implementation and it was suggested that this
discussion be postponed until the new developer from Dublin is available.
3. SRM2 Testing
Nick commented that some of the Tier-1 sites according to their schedule will deploy SRM2.2 by the end of this
month. Discussion with Stuart and Philippe resulted in the idea to ship the upgraded LCG Utilities such that SRM2 can
be queried first if available. For uploading output data this becomes difficult because space tokens must be defined in
the CS. Philippe queried whether we should change the -tape and -disk definitions to have one for each space. These
tests should be performed now with the current production tools in DIRAC2.
Latest version of LCG Utilities should be installed at the Tier-1s. There is no mechanism to query the version of LCG
Utilities installed at the Tier-1 sites so this should be investigated. ( ACTION: Roberto ) Nick commented that SRM2
tests failed at GridKa due to the required gLite version not being available. Andrei requested to gather all necessary
information and discuss at the Thursday operations meeting.
4. Production Operations Issues
- Release v2r18
Andrei reported that the job deletion logic is now fixed and secured in v2r18. Packaged with this release will be
lhcb-proxy-init that properly initializes the proxy via dirac-proxy-init and can upload the credential to the myproxy
repository. Joel commented this could be added to the LHCb startup scripts. Matvey will commit this to CVS. ( ACTION:
Joel, Matvey )
- Using VOMS proxies ( roles or groups )
After discussions at the Operations meeting it was agreed to abandon the notion of groups in LHCb and only express
use-cases and LHCb user groupings through VOMS roles. Nicknames will be stored as VOMS GAs. The VOMS role
should become a parameter of Jobs in the future and the CS will be updated from VOMS regularly. The VO ID Card will
be updated today such that redundant groups can be removed. ( ACTION: Joel )
Roberto commented that there are different possibilities at registration time from the VO Manager to insert new users.
VOMRS will be able to propagate changes of DN to the LFC via the immutable 'nickname' field but this is not currently
available and should be investigated. Andrei remarked that an LFC proxy admin service can be exposed to Joel as VO
manager to perform and automate certain tasks.
- RAL and PIC data migration
Marianne reported no updates since yesterday, plans were added to a Twiki page. FTS job submission is ongoing. PIC
migration is scheduled in two weeks from now. The LFC people are not yet informed of the migration because the
dates are not yet fixed. ( ACTION: Marianne ) Raja commented that the migration timescale would be 10 days
assuming the FTS jobs proceed without delay.
5. Pilot Agents Usage
Andrei attached a draft document to the Agenda page. This clearly highlights the mechanism within DIRAC for this
and comments are welcome. Stuart commented that it may be worth explaining how the generic pilot proxy could be
secured in the glexec logging mode. This is not an issue for the set UID mode. ( ACTION: Andrei )
6. DIRAC3
- Workload Management status (job submission, optimizers, monitoring)
Andrei mentioned that basic monitoring and submission tools are available. Stuart commented that progress had been
made with the optimizers. A simple API version is being prepared now in order to test the full chain. The monitoring
webpage is available in a preliminary status and Matvey is continuing with this in the new web framework.
- Data Management status
The LFC client has been finalized and ported to DIRAC3. Prototypes of the Storage classes for e.g. SRM2 are available.
The Storage Element class will be heavily reimplemented (originally a DIRAC2 Review recommendation). After this
the Replica Manager will be started.
- Production Management status
Gennady is working on a Repository Database and is prototyping the commands for submission of production
workflows. The API will still be used to submit the jobs to the WMS.
- Release tools
Tools for making releases starts to become urgent when testing job execution and submission etc. Joel (and Ricardo
when available) will try to proceed with this.
7. AOB
Philippe proposed a table for DIRAC3 development goals and proposed timescale, Andrei agreed to prepare this.
There are minutes attached to this event.
Show them.