Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
2-R-014 (CERN)
2-R-014
CERN
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 3rd March 2009, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: P.Charpentier, J.Closier, Z.Mathe, S.Paterson, M.Rashid, V.Romanovsky, R.Santinelli, A.C.Smith, F.Stagni
Phone: R.Graciani, A.Tsaregorodtsev, A.Zhelezov
Apologies: M.Sapunov
0. Ongoing Action List
(57) - 24/06/08 - Vladimir,Roberto - Automatic publishing of BDII / GOCDB information in the CS using GOCDB site names and provide a summary of recent changes
(83) - 01/09/08 - Stuart,Roberto - To perform glexec tests (pending SCAS certification)
(90) - 09/09/08 - Joel,Stuart - Use new ProductionOutputData JDL parameter in modules instead of creating LFNs
(93) - 16/09/08 - Andrew - Improve RAW data monitoring tools and status of files in online RunDB
(94) - 23/09/08 - Andrew, Marcos - Using SystemLogging service for dynamic services status monitoring (Andrew to provide FTS example)
(95) - 23/09/08 - Roberto - SLS and SAM sensors development, upgrade SE tests with new DMS unit test
(96) - 23/09/08 - Roberto - Update Dashboard SAM tests visualization, GOCDB / DIRAC site name mappings
(109) - 14/09/08 - Matvey - Expose DataLogging interface on the web page (pending lhcbweb deployment)
(111) - 14/09/08 - Alexei,Andrei - Productions with file merging and associated data distribution implementation
(116) - 18/11/08 - Ricardo - Circulate a report on the recent Gauss CPU studies, steer the CPU related activities in DIRAC
(122) - 13/01/09 - Ricardo,Vladimir - Develop specialized SAM Agent to track and report errors with regard to pilots
(124) - 20/01/09 - Stuart - Apply dataset limit sanity check for user jobs (limit to 100) to client side e.g. LHCbJob
(126) - 20/01/09 - Andrei - Document the design of the workflow / production management components
(132) - 03/03/09 - Stuart - Update LFN construction utility to store production output directories as a parameter
(133) - 03/03/09 - Matvey - Provide request DB interface page
(134) - 03/03/09 - Matvey,Andrew - Provide RunDB monitoring page
(135) - 03/03/09 - Andrew - Send accounting from Online DIRAC instance via the gateway to PIC
(136) - 03/03/09 - Vladimir,Roberto,Joel - Integration of existing SAM infrastructure tests into DIRAC SAM framework
Completed Actions:
Done - (106) - Ricardo - Produce twiki documentation for dirac-install
Done - (113) - Ricardo, Adri - Document the CS procedures for both the mechanism to define authorization rules in DIRAC and also the current group properties
Done - (120) - Ricardo - Update TQ Director to deal with a mixture of generic / private pilots at sites (and document the submission policy)
Done - (127) - Andrei,Adri - Multiple DNs in DIRAC for use-case of DN change
Done - (129) - Ricardo - Document the job prioritization mechanism
Done - (130) - Philippe,Andrew,Stuart - Prepare proposal for the DIRAC release procedure
Done - (131) - Andrei, Stuart - Test BK query for production input data
1. Action Updates
- Action(57) - Automatic publishing of BDII / GOCDB information in the CS
Vladimir reported that CS information is filled via a script, this will be put into an agent as soon as possible.
- Action(83) - gLexec tests
Ongoing, hope to make progress end of this week / next week. Roberto commented that currently SCAS is being
restarted every 30mins due to a globus memory leak but this should be transparent for us.
- Action(90) - Production Output Data parameter
Stuart mentioned that the new finalization mechanism is in production, this splits the old job finalization into 4
modules: SendBookkeeping; UploadOutputData; UploadLogFile and FailoverRequest (documentation is being prepared
for this). The last place to update to the new utility is the BK report module, Stuart and Joel will try to update this in
the coming days. After some discussion it was agreed to discuss offline the possibility of storing output data LFNs per
job in the production management system offline.
- Action(93) - Improve RAW data monitoring tools and status of files in online RunDB
Andrew reported that a service is on the gateway allowing to query for the status of individual files. Introducing a run
by run breakdown of files is the next step and will be pursued with Matvey. The interface to the request DB is already
in place but would also be advisable to have a web page for this.
- Action(94) - SystemLoggingDB example
Discussed last week with Marcos, for the FTS error case the classification seems to be easy. Relating the error to the
source / destination requires a bit of code and it seemed feasible for the developer providing the error to be
responsible for this. Ongoing.
- Action(95),(96) - Service Monitoring
Roberto made an overhaul of the SAM infrastructure tests. The SAM test availability numbers for the last quarter
appear to be consistent with known problems at the Tier-1s. Stuart queried whether the recent 'unavailable' files
problem was visible in these tests, Andrew confirmed this is the case in the latest version and Roberto will update the
suite accordingly. The Dashboard GUI will be updated to display DIRAC site names after Pablo returns from the EGEE
conference.
- Action(109) - Expose DataLogging interface on the web page
Ongoing. This is not yet deployed in production.
- Action(111) - File merging productions
No progress here, the recent efforts have been on the production request GUI.
- Action(116) - CPU studies
Ricardo included publishing of the normalized length of the queue in the pilot when requesting jobs. The next step is
to publish a benchmarking factor in the pilot and to redefine the units of CPU that we use.
- Action(122) - Develop specialized SAM pilot director to submit, track and report errors with regard to pilots
Pilot submission / aborting errors can now be propagated to the SAM DB. Vladimir will write an agent to track the
SAM jobs that don't start and report to the SAM DB.
- Action(124) - Apply dataset limit check
Ongoing with low priority.
- Action(126) - Document the design of the workflow / production management components
Andrei added some further documentation to the production procedure page. The production request description is
ongoing.
- Action(132) - Ricardo - Provide example of stalling Gauss jobs to Gloria
Gloria has managed to reproduce the problem from Ricardo's mail.
2. Production Status Report
Moved from Operations, integration of existing SAM infrastructure tests into DIRAC SAM framework ( ACTION:
Vladimir,Roberto,Joel ).
- Report from the Production meeting
The dummy activity is on hold pending fixes to Gauss. Production cleaning policy, urgent to have a status 'Archived'
that makes old Completed productions disappear on the web page. How to treat productions after they have been
completely removed is also an issue (~300 of these in the system). The state machine will be better described in the
future. Roberto mentioned that a mega-patch is being applied to the CERN WMS instances in the coming days.
The new machines requested by Joel are being prepared, these have 2 disks and increased system memory. Several
optimizations can be performed at the MySQL level as soon as this becomes available. Although we are currently
requesting 16GB of memory it was made clear by Andrei that this requirement will not seriously increase in the long
term.
volhcb06 is currently full, the next dummy production will go to the 'test' namespace.
3. DIRAC Tasks
- Framework status
Ricardo described the recent problems with the python subprocess module that resulted in some stalling jobs. The
latest version of python2.4 may have fixes for this problem but this is being investigated. The Watchdog CPU check
was simplified and is under testing in development. The python2.5 bindings are still not available for the LCG utils /
LFC / gfal, Roberto will chase this up.
- Bookkeeping status
The opposite method of getAncestors (getDescendents) has been implemented by Zoltan. The DQ flag command line
scripts have been provided in time for FEST, Zoltan will prepare documentation for the DQ shifters.
- WMS status
Several fixes mentioned above.
- DMS status
Andrew has given up finding a new working gfal version and is pending a response from the developers.
- Production Management status
There was some discussion regarding the GUI, Alexey will try to implement the fixes / suggestions from the feedback
of the DIRAC week.
4. AOB
Andrei reminded those round the table about the proximity of the CHEP conference. Philippe suggested that the
rehearsals could take place during the next SW week.
There are minutes attached to this event.
Show them.