Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/cgi-bin/conference?access_code=0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 19th June 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, S.Paterson, R.Santinelli, A.Smith
Phone: R.Bernet, N.Brook, A.Carbone, F.Donno, R.Graciani, G. Kuznetszov, U.Marconi, R.Nandakumar, M.Sapunov, A.Tsaregorodtsev
1. Production Status Report
Minutes from the production meeting are available here.
Joel and Roberto conveyed that almost 100 GGUS tickets had been submitted to rectify software installation and
mapping problems. As was discussed at the production meeting, the limit of CPU was reached for the Dimuon
production. This will be solved via reducing the number of events per job.
In order to prevent accidental submission of production jobs to the user system it was agreed to ensure
ProductionEnv only sets up the production and test systems. In parallel, the repository agent should also ensure that
the user mode is not authorized. ( ACTION: Joel, Gennady )
Ricardo reported that the WMS cleaner is now in place, jobs older than 3 months will automatically be accounted and
purged from the system. Roberto confirmed that the request for 1 month VOMS extensions had been made and was
awaiting approval.
The Castor intervention at CERN was successful and Joel mentioned that both PIC and CERN were performing well
with the reconstruction / stripping activity. Jobs eligible to GRIDKA and NIKHEF have failed with 'Exceeded max
staging retry' and are to be investigated. ( ACTION: Joel )
The recent staging problems at GridKa are under investigation and after consultation with the SRM experts Roberto
conveyed that the SRM needs some configuration / tuning. Another difficulty at GridKa is related to NFS access under
high load, ~500 concurrent jobs results in problems accessing the shared area. Joel agreed to remind Johan and
Michael about the urgent SRM problems. ( ACTION: Joel ) Stuart confirmed that the timing data for all Tier-1 sites is
being stored by the Stager and Andrew is to begin work on visualizing this information. ( ACTION: Andrew )
Stripping jobs have been running successfully at CNAF where no reconstruction is pending, more jobs are to be
submitted there. Jobs at RAL have completed successfully and several are still staging. At IN2P3 some jobs have
been successful and others have failed, this is under investigation. ( ACTION: Joel )
With regard to the SAM tests, Roberto supplied an SE test that Joel is starting to add to the framework. The SE test
will be run later in the week and if successful could move to the ops critical test suite. A mechanism to check the
mapping of credentials is pending. ( ACTION: Joel )
2. Production Operations Issues
A DPM Storage Monitoring Proposal is available on the Agenda page by M.Ciricello and F.Donno. This triggered some
discussion about storage and infosites discrepancies (~10% difference when not metric). It was agreed to report to
Flavia's group that consistent behaviour everywhere is desired. ( ACTION: Andrei ) To ensure an accurate
measurement of disk usage, Andrew suggested waiting for the DC06 cleanup to be over. ( ACTION: Andrew )
Philippe requested a summary of the storage pledges versus deployed amounts per Tier-1 site and Nick was asked to
circulate the wiki page for LHCb storage requests in 2007. ( ACTION: Nick )
Raja mentioned that Transfer Agents have been hanging on the VO-boxes. This is related to the retrieval of an invalid
proxy from the WMS.
Gianluca reported that proxy renewal was now working, yet still using the Perl script. It was agreed to convert the
Perl script to Python and incorporate the functionality in the LCG utilities. ( ACTION: Gianluca ) Andrei agreed to test
the recent changes and immediately deploy these fixes to the production system. ( ACTION: Andrei ) A bug was
reported recently which involved the creation of temporary directories that were not cleaned during job submission.
Gianluca was asked to ensure that temporary directories were cleaned and that the gLite Agent Director module
selects a smaller number of jobs per loop in order to reduce the load observed by Ricardo. ( ACTION: Gianluca )
Roberto confirmed a mail was sent regarding the gLite RB instances at CERN and their deployment but there was no
news at the time of the meeting.
Stuart reported that half of the tests for the Job Sanity Agent were in place. Deployment to the Test system is
expected by the end of the week. Andrei suggested restricting the output data check to production jobs and this was
agreed. Below is a summary of the job sanity agent functionality:
- Checking for output data already existing in the LFC
- Failing jobs with problematic JDL (not existing or illegal)
- Checking that input sandboxes are correctly uploaded (non-zero size)
- Failing jobs with too much input data specified (e.g. jobs with > 200 files)
- Failing jobs with input data not specified according to LHCb conventions (e.g. not /lhcb/ or containing multiple
slashes)
( ACTION: Stuart )
Ricardo raised the issue of Stager progress monitoring. Stuart mentioned that the 'failed' 'Exceeded max staging
retry' final job state contained the timing information and list of problematic LFNs which failed to be staged. The
current strategy is to fail jobs for which input data takes greater than 24hrs to become staged. Joel mentioned that
extracting this information from the monitoring webpage job by job was time-consuming. It was decided to provide a
DIRAC script to gather this data from the JobDB and present it in a form suitable for the submission of GGUS tickets
etc. ( ACTION: Stuart )
3. DIRAC3 Tasks
Ricardo mentioned that some problems had been found with the frameworks when under heavy load. Investigations
are ongoing. ( ACTION: Ricardo, Adri )
A report is being generated for PIC regarding support and hardware etc. Ricardo agreed to circulate this document for
comments when available. ( ACTION: Ricardo )
There is a DIRAC3 workshop next week and it was decided to retain the PASTE and production meetings.
4. AOB
Joel wished to clarify the user accounts required on WMS machines and Andrei confirmed that the jabber user is no
longer required, only dirac and mysql are necessary. The owership of files on volhcb03 will be modified accordingly.
( ACTION: Joel )
There are minutes attached to this event.
Show them.