Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 22nd July 2008, CERN
Chair: S. Paterson
Minutes: S. Paterson
CERN: Z.Mathe, S.Paterson, V.Romanovsky, R.Santinelli, A.C.Smith
Phone: M.Bargiotti, R.Graciani, M.A.S.Miguelez, R.Nandakumar, M.Sapunov
Apologies: N.Brook, A. Tsaregorodtsev
0. Ongoing Action List
(01) - 06/11/07 - Marianne, Joel -Remedy input data not available and catalog access issues then update DIRAC2 ProcDB
(07) - 27/11/07 - Zoltan, Elisa - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos - Develop a general service monitoring framework leading to a virtual control room
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(57) - 24/06/08 - Vladimir - Automatic publishing of BDII information in the CS using GOCDB site names
(58) - 15/07/08 - Marianne - Clean up extra 3M events for production 2146
(59) - 21/07/08 - Ricardo - Add ldap python modules to DIRAC python distribution
(60) - 21/07/08 - Vladimir - Develop tools to check stalled jobs for outputs and clean up as required
(61) - 21/07/08 - Andrei - Investigate request generation problems in the workflows
(62) - 21/07/08 - Roberto - Check instabilities in production gLite WMS instances
(63) - 21/07/08 - Roberto - Develop DIRAC3 script for adding new users to the VO
(00) - 21/07/08 - DIRAC developers - DIRAC3 (ongoing)
1. Ongoing Action Updates
- Action(07) - Bookkeeping
Zoltan confirmed that the agent to synchronize new and old bookkeeping is running smoothly.
- Action(55) - Recover data in BK DB
Marianne reported that the change to have at least one ancestor file to alllow the processing of existing Bookkeeping
requests has not made a significant difference in the amount of pending requests. The AMGA server recently became
stuck and this created a lot of errors but after restarting some components this was resolved.
- Action(57) - Publishing BDII information
DIRAC3 CS with BDII information is pending the ldap / pyXML modules. Ricardo agreed to add these modules to the
DIRAC python. ( ACTION: Ricardo )
- Action(58) - Clean up 3M event 2146
Marianne reported no progress on this so far.
2. Production Status Report
Stuart gave a summary of the relevant issues discussed at the Production Operations meeting:
- Input sandbox LFN for WorkflowLib
Many jobs for recent productions ( ~3.5K) have failed while downloading the input sandbox LFN corresponding to the
WorkflowLib. An segmentation fault in GFAL occurs while attempting a download from GridKa and this should be
investigated. Due to the instabilities of GFAL (and the Tier-1 SEs) we may have to consider downloading and shipping
the workflow libraries with the jobs in the future.
- NIKHEF staging problem
The nearline / online issue at NIKHEF still persists after many weeks. It was agreed to temporarily perform a gfal.get
on our side instead of a gfal.prestage but still to pursue the resolution of this problem. Ricardo mentioned that
passing the wrong parameters in the gfal dictionary results in an unexpected exception later in the code (thread
safety issue) observed with the thread safe version of the library. This issue will be escalated again.
- Job history agent not running in production
Ricardo mentioned 2 days of fine tuning is required before releasing the current development DIRAC version with the
new proxy infrastructure. It was agreed to wait for this to resolve the job history agent issue.
- Stalled jobs
Develop tools to check stalled jobs for outputs and tidy submitted jobs. ( ACTION: Vladimir )
3. DIRAC3 Tasks
There is an outstanding issue of proxies becoming mixed in the gLite WMS affecting the retrieval of the outputs.
There was a request from the gLite WMS developers to perform some tests to isolate whether the problem is in proxy
renewal mechanism of the gLite WMS.
Ricardo reported that the development version of DIRAC with the new proxy infrastructure is almost ready to go.
Tested with ~20/30k jobs / day and it was observed that the Matcher and JobStateUpdate services were putting most
of the resulting load on the system. Some connections are not properly closed and Ricardo is testing whether the
timeout on RPC calls is affecting this. With the Matcher the main issue is the initial selection of the jobs in the DB
when under load. It was agreed to test the production management infrastructure prior to release. Request
generation in the workflows is still to be fixed. ( ACTION: Andrei ) Failover should be fixed prior to the production
release. Hopefully put into production by the end of the week. Combined software installation will also be put into
production at this point.
Several new accounting views are available in production and JobHistory information is also sent to the accounting.
Some gLiteWMS instances were observed to be unstable after recent tests from Ricardo, it was agreed to investigate
these RB instances. ( ACTION: Roberto )
Roberto mentioned the script used by Joel to register a new user in the VO was not fulfilling all use-cases. Stuart
suggested that the script is rewritten in python and added to the DIRAC3 LHCbSystem, new users can then
automatically also be added to to the DIRAC CS. ( ACTION: Roberto )
Stuart mentioned that the rewritten SAM test suite in DIRAC3 will start to be tested now using the Development
installation.
4. AOB
Since Andrei is available Monday -> Wednesday next week, it could be possible to have a DIRAC3 week (28th July)
this will be confirmed offline.
There are minutes attached to this event.
Show them.