Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 27th November 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, A.Casajus, P.Charpentier, J.Closier, E.Lanciotti, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, G.Kuznetszov, M.A.S.Miguelez, R.Nandakumar, A.C.Smith, A.Tsaregorodtsev
Apologies: G.Castellani
0. Ongoing Action List
23/10/07 - Marianne - Request for basic documentation for restarting the Bookkeeping services
23/10/07 - Marianne - Bookkeeping should be migrated from SLC3 -> SLC4 and installed on volhcb05
30/10/07 - Marianne - Check number of events for prod 1957 in BK against LFC
30/10/07 - Marianne - Update of LFC replica flags (C checked, N checked but not present)
06/11/07 - Gianluca, Marianne - Remedy the Exceeded Max Staging Retry jobs and update ProcDB as required
06/11/07 - Gianluca, Marianne - Agents for automatic resolution of IntegrityDB entries should be developed
13/11/07 - Roberto - Contact ATLAS representative to understand VOMS usage in other VOs
13/11/07 - Alexei - Investigate recent sporadic failures at GridKa for production jobs
13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
27/11/07 - Roberto - Redesign LFC intervention plan for upgrade to latest version
27/11/07 - Andrei - Prepare draft pilot document as internal LHCb note
27/11/07 - Marianne - Investigate discrepancy of events for productions 1960 and 1962
27/11/07 - Elisa, Marianne - Prototype bookkeeping with API wrapped in DISET service
27/11/07 - Andrew - Make proposal for CS SRM2 SE definitions
27/11/07 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - 06/11/07 - Raja - Install v2r17p2 on VO-boxes
Done - 13/11/07 - Andrei - Preparation of the pilot agent framework document
Done - 23/10/07 - Ricardo (Andrei) - Update Cleaning Agent to prevent user WMS problems
Done - 13/11/07 - Joel - Check dump from Marianne and update ProcessingDB
Done - 13/11/07 - Alexei - Produce dump of file namespace at GridKa for Marianne to proceed with LFC replica flags there
Done - 13/11/07 - Roberto - Contact site administrators for Tier-1s and install latest version of LFC with Oracle streams enabled
Done - 13/11/07 - Roberto - Check status of LFC admin interfaces with WMS instances as trusted hosts
Done - 13/11/07 - Roberto, Gianluca - Resolve issue with Raja's credential causing failed pilot submissions
Done - 13/11/07 - Joel - To check whether ancestor files are required for reconstruction production 1981
Done - 13/11/07 - Roberto - To check the LCG Utilities status for SRM2 at the Tier-1s
Done - 13/11/07 - Joel, Matvey - lhcb-proxy-init should be committed and added to the LHCb startup scripts
Done - 13/11/07 - Joel - To update VO ID Card to remove obsoleted group definitions
1. Production Status Report
Minutes from the production meeting are available here.
Joel reported that all requests have been submitted but a problem with the Bookkeeping prevented the production
managers from checking the status. Marianne commented that after a restart of AMGA the server was stuck.
Productions 1960 and 1962 seem to have a discrepancy, Marianne agreed to investigate. ( ACTION: Marianne )
The stripping workflows are now using the correct Brunel version. Test jobs to CNAF had failed with the account
mapping and shared area usage. SARA was in scheduled downtime and as a result NIKHEF is still banned. CERN and
PIC were running jobs smoothly. IN2P3 has recently been experiencing problems and will be in downtime until this
afternoon. Jobs are pending for RAL and a mixture of results have been observed at GridKa.
As part of the RAL migration, transfers have started and Marianne is starting to submit the jobs. There was some
discussions over problems transferring certain files. Inconsistencies between LFC and SE are being observed for the
DIGI files. This activity is ongoing. ( ACTION: Marianne )
2. Production Operations Issues
- Bookkeeping interface update
Elisa is developing a python client based on the AMGA python API to replace the web page java scripts. Philippe
reminded that the BK working group indicated that the web page would be kept for accounting information and not
primarily for users. It was agreed to build a DISET service based on Elisa's module that could be extended initially to a
CLI and could eventually be used to interface to the DIRAC3 web page framework. ( ACTION: Elisa ) Marianne will
investigate the BK Manager that currently depends on javascript, this should be extended to DISET. ( ACTION:
Marianne ) In order to prototype a test machine should be setup. Andrei suggested using volhcb01 after the
migration of user and BK systems.
- PIC migration
Marianne has started to initiate the migration and has contacted the LFC support. Adri reported that everything
should be ready to start the NStore migration at PIC side by the end of the week. The timescale for SRM2 migration is
less clear but Adri confirmed this should occur by the end of the year (or early next year).
- SRM2 Testing
Andrew reported a summary of the SRM storage workshop. When recalling files from tape, dCache doesn't know
where to put the file. It appears that a space token must be supplied now for both dCache and Castor whenever
recalling from tape. Issuing 'gets' on disk, dCache will ignore the space token but Castor does require it. The best
policy for our usage in the future would be to always supply a space token regardless of whether the backend storage
uses it. Philippe will raise the issue at the MB. ( ACTION: Philippe ) Therefore, it was agreed that the space
management must now be included in DIRAC. Andrew reported that the space token can be stored in the LFC as a
metadata entry if required and separate logical SEs for each space token will have to be defined in the CS. Andrew
agreed to circulate a mail highlighting recent discussions with Flavia on SRM2 and make a proposal for how to handle
this in the CS. ( ACTION: Andrew )
The issue of testing SRM2 for TURL resolution on the WNs and reading by the applications will be postponed until next
week. The developers will make every effort to ensure DIRAC3 can be used for this. ( ACTION: Stuart, Joel )
- VOMS proxies, sgm and production accounts, LFC admin interface
v2r18 is now released in production and this deals consistently with VOMS proxies everywhere. Roberto will update
the intervention plan for the migration to the new LFC service. ( ACTION: Roberto ) During the migration, the CNAF
instance will be used as a RO instance and the CERN master and RO instances shall be upgraded. Then the CNAF LFC
will be removed from the allowed list.
Any problems with widespread use of VOMS credentials for the production managers, such as accessing LFC for
writing, can be identified by Raja while monitoring the VO-boxes. ( ACTION: Raja )
There was some discussion on the use of static sgm accounts versus pool accounts. In summary, we have no
preference as long as the implementation works.
- volhcb05 migration, BK and user system
Andrei wondered whether the port-scan had been performed and Joel confirmed this was the case. The migration to
volhcb05 can proceed. In order to avoid migrating 10s of GB of sandboxes it was suggested to schedule the draining
of the system from Friday midday. Outputs are expected to be retrieved by Monday midday and Monday afternoon
the migration can proceed.
For the BK services, Marianne has installed the services on volhcb05 but there are problems with LFC access, this
should be certified before migration occurs, ongoing action.
3. DIRAC3
- Workload Management status (job submission, optimizers, monitoring)
Stuart reported that the CE classes are in place. Work is proceeding with the Job Agent.
- Release tools
Installation and setup scripts have been prototyped by Joel.
- Production Management status
In parallel the development of production job submissions can continue.
- Service Monitoring
Adri reported that the DIRAC3 services are being updated after a recent pyOpenSSL upgrade. A service monitoring
framework is under development.
- Data Management Status
TransferDB with Agent to submit and monitor FTS jobs in place. Strategies are being compared for replication.
4. AOB
Andrew raised the issue of BK migration and wondered whether there was a plan for upgrading this to DIRAC3
eventually. It was agreed this should be first done in DIRAC2 before performing the simple upgrade to DIRAC3.
There are minutes attached to this event.
Show them.