Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 24th April 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, N.Brook, G.Castellani, A.Smith, S.Paterson, R.Santinelli
Phone: R.Bernet, A.Casajus, C.Cioffi, R.Graciani, U.Marconi, M.A.S.Miguelez, R.Nandakumar, A.Tsaregorodtsev
Apologies: P.Charpentier, J.Closier
1. Production Status Report
Minutes from the Monday meeting are available here.
Simulation:
=========
Raja confirmed that there is no news as all pending requests have been satisfied.
Reconstruction:
============
Stuart reported that reconstruction jobs are now running smoothly at CERN, CNAF, IN2P3, PIC and RAL. Only a small
number of jobs were sent to GridKa with mixed results. Recent issues with NIKHEF include: storing the output rDST
file at the local storage; LFC access from WNs; inconsistent lcg-gt behaviour on the WNs. A GGUS ticket should be
submitted to report this ( ACTION: Stuart ). In order to efficiently spot such SRM instabilities it was agreed to put
SRM tests as part of the SAM tests for now ( ACTION: Joel ).
The next step is to amp up bb-maxi reconstruction and restart the bb-inclusive from scratch. Marianne is half-way
with preparing the bb-inclusive and should finish by Wednesday at the latest. The Processing DB should then be
populated with these files ( ACTION: Marianne ). Roberto commented that he is working with Marianne to access the
CNAF RO Oracle backend directly ( ACTION: Roberto, Marianne ).
We have subsequently reached a new record of 500 concurrent reconstruction jobs at the Tier-1 sites.
Stripping
=======
Several Stripping test workflows had been submitted to the Test system without success. A problem has been
identified with Brunel execution after the stripping phase that seems to be Gaudi related and the work to fix this is
ongoing. Ricardo offered to run a test job at PIC to see if any problems can be spotted locally and check the CPU
timings over a larger number of events, these can be given to Patrick for verification ( ACTION: Ricardo ). In parallel it
was agreed for Joel to construct workflows for all sites which are currently stable for reconstruction. ( ACTION: Joel )
For the dCache under high load tests, Adri reported there were many false-positives from the initial test and after
some new modifications this is being rerun. Adri suspects that errors occur during upload of files when the pool is
disabled. This is a standard operation for dCache but seems to cause problems occasionally. As soon as more
evidence is available the developers will be involved but this suggests that the failures are not dependent on high
loads. ( ACTION: Adri)
2. Production Operations Issues
Stuart summarized the current Stager Agent status, this is now in robust operation after several optimizations for
scalability. Ricardo mentioned that jobs in the 'staging' status are difficult to track by site. Stuart agreed to put this
development into the Data Optimizer. ( ACTION: Stuart )
The second half of Andrew's detailed plan for the DC06 cleanup was:
1a) Extract list of LFNs for '.sim'/'.digi' produced with incorrect versions (Carmine)
1b) Extract list of LFNs for rdsts which have LFNs in the above list as ancestors (Carmine)
2) Produce list of PFNs for all the above LFNs (Marianne)
3) Remove these replicas from the catalogues (LFC, Bookkeeping) (Marianne)
4) Generate by-site list of PFN/directories to be removed and contact sites (Marianne)
Marianne has started removing SIM files so the cleaning operation is underway. All data is at CERN except for
problematic data files which could not be migrated (i.e. may not exist). The disks at the Tier-1 sites must be cleaned,
Marianne is compiling a list of all files for removal from the LFC which should be ready by the end of the week. The
LFC developers are then to be contacted with the list of files and an intervention should be planned ( ACTION:
Marianne, Roberto ). After this the local site administrators can be contacted for the physical removal of these files. It
was agreed that for the files affected by the dCache bug, any replicas should be transferred to CERN and non-existing
replicas should be removed from the LFC ( ACTION: Andrew ). Last week's Action regarding LFC LFN registration
failures carries over to next week ( ACTION: Andrew ).
Gianluca is continuing to modify LCG.py and is testing it with affected WMS components ( ACTION: Gianluca ). After
successful testing, the User system can be migrated to volhcb01 and we can announce to the experiment the switch
to VOMS proxies. ( ACTION: Andrei )
Stuart reported some initial tests of glexec at Lyon have been successful. The next step is to write a DIRAC
computing element to make use of this. ( ACTION: Stuart )
3. LHCb Policies
Nick mentioned that Ulrik supports the removal of the Ganga LCG backend. This is currently delayed pending the
next Ganga release which allows submission of Root scripts via DIRAC.
There was some discussion of VOMS nicknames, it was mentioned that once these are available, it would be possible
to create a unique user namespace with correct ACL in the file catalog.
For user data storage on the Grid it was agreed to make the change in the job wrapper to store output data at the
Tier-1 associated to the job.
4. DIRAC3 Tasks
Due to time constraints, this section was cut short. Andrei has uploaded revised versions of the DIRAC CS schema
and the DIRAC DB schema to the agenda (link to documents) and all comments are welcome.
There was some discussion of the Site Mask for DIRAC3 and it was agreed to tie this to the job types in the future.
After extended discussions regarding how to keep track of changes to the CS. It was agreed for Ricardo and Adri to
think about this and make a proposal ( ACTION: Ricardo, Adri).
5. Top 5 Issues
Nick attached the LHCb Top 5 Issues presentation to the agenda page (link to slides).
6. AOB
The tentative agenda for the next Software Week is available and it was mentioned that Marianne, Gianluca, Roberto,
Andrew and Stuart will be at the EGEE User Forum during this week.
There are minutes attached to this event.
Show them.