Production and PASTE Meeting, 10th July 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, S.Paterson, R.Santinelli, A.Smith
Phone: R.Bernet, N.Brook, F.Donno, R.Graciani, G.Kuznetszov, U.Marconi, M.A.S.Miguelez, A.Tsaregorodtsev
Apologies: J.Closier, R.Nandakumar
1. SRM2.2 Tests
Nick attached a
SRM2.2 testing proposal to the agenda page. Nick agreed to generate a summary of the discussions with
actions and responsibles (included below):
LHCb SRM 2.2 task list
(i) Identify 15 TB of DIGI files at CERN ( ACTION: Nick )
(ii) Use of space tokens withi DIRAC DMS for FTS2, including small scale tests - timescale next DIRAC
meeting (17th July) - ( ACTION: Andrew )
(iii) Configuration of FTS channels (ACTION: Flavia)
(iii) Pursue issues associated with lcg util needing BDII also pinning, file lifetime etc - missing functionality
through lcg utils - (ACTION: Flavia)
(iv) Move the current functionality of stager to SRM 2.2, including small scale tests - next DIRAC
meeting (17th July) - ( ACTION: Stuart )
(v) New functionality of stager with SRM 2.2 - "pinning","releasing",...:
- preparation work associated with new plug-ins - ( ACTION: Andrew )
- full pinning etc functionality (relies on lcg utils) - 3/4 weeks - ( ACTION: Stuart, Andrew )
(vi) Workflow for processing jobs - Joel when he returns from holiday (2 weeks)
2. Production Status Report
Minutes from the production meeting are available
here.
There were many jobs seeming to fail at IN2P3 however this is due to the XML-RPC sandbox upload problem and the
jobs were in fact successful. Andrei agreed to mark those jobs correctly and investigate the XML-RPC error. ( ACTION: Andrei )
Many jobs at GridKa are failed with exceeded maximum staging retries. The recent site downtime at GridKa affected
those jobs but it is still to be understood whether the problems are resolved.
The disk storage at PIC is full and this is causing many transfer failures. Ricardo confirmed that all the Transfer
Agents were running normally. It was agreed for Marianne to check the transfer requests such that when replicas are
safely on disk storage they can be removed. ( ACTION: Marianne )
Stuart reported that the Job Sanity Agent was deployed to the production system. Production managers can now
reschedule stalled and failed jobs except for stripping (where the failures are under investigation). A script should be
provided for production managers to do this and a reasonable delay of ~1 week should be enforced. ( ACTION: Andrei )
3. Production Operations Issues
Stuart and Roberto reported the software installation status. Almost 30 GGUS tickets were submitted to sites, the
majority of these related to ownership of files in shared areas. Site admins were asked to clean up these files and
declare the shared areas as group writable. For three of the sites the VOMS mappings were not correct and a small
percentage had insufficient disk space available for the software. Stuart mentioned that results from SAM jobs are
often not published and this should be investigated. ( ACTION: Roberto, Stuart )
Roberto confirmed that the request for longer VOMS proxy extensions was refused. The gLite RBs can renew proxies
in the site batch queues so this should not be a problem.
Marco Cattaneo sent an email regarding CASTOR_DEBUG output not being uploaded. Raja's account mapping should
be checked. ( ACTION: Roberto )
There was some discussion of database updates on disk pools and the regularity of these. Ricardo suggested spreading
the load over the day instead of performing these operations all at once. Philippe agreed to raise the issue at the next
IT meeting. ( ACTION: Philippe )
Data requested by Matt Needham was accidentally removed from Castor by Marianne. Together with Carmine,
Marianne is searching for some similar datasets which can be distributed according to his needs. Marianne reported
that the CERN_Castor file cleanup is ongoing and she is preparing the lists for this. ( ACTION: Marianne )
4. DIRAC3 Tasks
A brief status summary was given around the table:
- Ricardo: still working on the accounting server
- Andrei: absent last week
- Stuart and Andrew: swamped with operational issues
- Gennady: working with Carmine to understand the Bookkeeping
The next DIRAC3 week is in 13th August.
5. AOB
Ricardo mentioned that the information on the LHCb VO Card was incorrect. A requested version of libshift on SLC4 is
incompatible with currently running versions. Philippe suggested that this information should be removed from the
VO card as we ship this library ourselves. ( ACTION: Roberto )
There are minutes attached to this event.
Show them.