Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0165202
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0165202
Production and PASTE Meeting, 8th July 2008, CERN
Chair: S. Paterson
Minutes: S. Paterson
CERN: M.Bargiotti, P.Charpentier, V.Romanovsky, S.Paterson, A.C.Smith
Phone: R.Bernet, R.Graciani, M.A.S.Miguelez, R.Nandakumar, M.Sapunov
Apologies: N.Brook, J.Closier, R.Santinelli, A.Tsaregorodtsev
0. Ongoing Action List
(01) - 06/11/07 - Marianne, Joel -Remedy input data not available and catalog access issues then update ProcDB
(07) - 27/11/07 - Zoltan, Elisa - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos - Develop a general service monitoring framework leading to a virtual control room
(55) - 08/04/08 - Marianne - Recover data that failed to be registered in the BK DB
(57) - 24/06/08 - Vladimir - Automatic publishing of BDII information in the CS using GOCDB site names
(58) - 01/07/08 - Roberto - Extract list of production role DNs for GGUS ticket submission
(00) - 10/06/08 - DIRAC developers - DIRAC3 (ongoing)
1. Production Status Report
- Update ProcDB - Action (01)
This action point is linked to the ongoing DC06 activity.
- New Bookkeeping system - Action(07)
Submission of XML files is being checked to old and new BK systems. Zoltan has implemented a BK file transfer agent
that runs on volhcb05 and triggers the update of the new Bookkeeping on volhcb07. Philippe raised the request from
Zoltan regarding the update of DIRAC3 with Ricardo (who has been having mail problems). Ricardo will prepare the
client installation on volhcb05. Elisa will update the DC06 configuration to more clearly reflect MC in the configuration
name. The next step is to make a more recent migration pass on the recent productions.
- Recover lost BK data - Action(55)
The missing BK XML files from over the past year have created a backlog that becomes difficult to work around (since
the XML requests are somewhere in a tarball on AFS). Marianne located ~1K files in the error directory of the existing
Bookkeeping failing because the ancestor input files were not already present. Philippe suggested to relax the
ancestor requirement for files in this state since any subsequently created jobs will fail. As long as one input file
exists the other ancestors can in principle be found.
- Automatic publishing of BDII information - Action(57)
Vladimir is progressing with this agent but requested the python ldap module be added to the distribution, Ricardo
agreed to look into this.
Philippe conveyed that there was not much DC06 activity over the weekend, Gianluca's agent is not functioning due to
the lack of integrity DB in DIRAC2. The same jobs are created / submitted in a loop with the same files being
recycled. The ProcDB was cleaned up after a synchronization with the LFC. Requests for simulation ~100K events
have been received yesterday and will be processed using DIRAC2. IN2P3 jobs have run successfully, reading all
input data files. There seems to be an integrity problem for several files at IN2P3 for which the jobs are terminal. For
old jobs in the assigned state in the ProcDB the outputs of the corresponding jobIDs are being checked and the files
are being reset as unused. This was run on the b inclusive 1896 & maxi bias 1932 productions and liberated ~200
jobs. The failover copies (removed from the LFC) are still in the ProcDB due to a problem in the updated logic.
Marianne has completed the removal of DC04 data in /grid in the BK and storage, ~150TB on CERN-tape was freed.
The files on castor but not in the LFC were also cleaned up. Philippe remarked that there were ~150K files outside of
the normal /grid namespace that should also be tidied. After noticing the standard namespace is slightly polluted with
files e.g. /grid/lhcb/* it was suggested to have a well-defined 'volatile' namespace for test files that is regularly
purged by data management tools.
2. DIRAC3 Rollout Checklist
At the time of writing most developers have made an effort to update the status of their assigned tasks. This is an
ongoing effort but should be kept up to date. Developers are asked to check the date at which the task was
completed. This can also be used to estimate the completion date before the task is declared finished.
Andrew clarified that the VO-boxes require a simple agent to forward requests to a central service. Raja can reinstall
the VO-boxes as soon as this is available. For each request type there will be a dedicated agent to process the related
requests.
The point about overriding the application status was raised by Ricardo. It was agreed to update the workflow
modules to correctly retain the first major error.
Full testing of the Failover was earmarked as a major milestone for the rollout of DIRAC.
Andrew commented that the BK plugin was tagged and fixed to ensure the 'has replica' flag is now correctly set.
3. CCRC operations issues
Raja raised the issue about failing transfers at CNAF and has opened a GGUS ticket for this issue. Philippe mentioned
that StoRM was down without advertisement and is not sure of the current status. IN2P3 will be down for 2 days next
week. Automatic management of the site mask is an ongoing issue that requires some attention in the future.
Migration of the SAM tests to DIRAC3 is also becoming more urgent. Stuart reported that work on this has started.
The job rescheduling policy with regard to input data failures was also raised, introducing a delay between
reschedules was suggested. A new DIRAC3 version will be deployed on volhcb03 for pre-production testing this
afternoon.
The recent issues with gfal and gfalthr have been pinned down to the type handling of the python binding. The
initialization results in the familiar segmentation violations observed during the CCRC. Andrew found that replacing
default values with empty strings solves the problem and will deploy this fix as a temporary measure. This issue has
been reported to the developers in order to improve the error handling. Ricardo also mentioned that the global
python environment was not thread safe (for both gfal and gfalthr).
4. AOB
None
There are minutes attached to this event.
Show them.