Production and PASTE Meeting, 8th December 2009, CERN

Chair:  A.Tsaregorodtsev
Minutes:  S.Paterson

CERN: J.Closier, S.Paterson, R.Santinelli, A.C.Smith, A.Tsaregorodtsev, F.Stagni

Phone:  R.Nandakumar, M.Sapunov

Apologies:  P.Charpentier, Z.Mathe


0. Ongoing Action List

#12165 - Roberto - no progress yet. Joel was asked to document a twiki with instructions to the operator for each alarm. 
#12383 - Resource status monitoring - it was agreed to aggregate all DIRAC information in one place such that it can be pushed to external sources as necessary
#12526 - The new production machines have been requested but not yet deployed. Testing will occur in the new year. Joel confirmed that the new LogSE is now available.


1. Production Status Report

- Current productions, LHC collisions data
Stuart summarised that the reconstruction of the collisions data was launched on Sunday and the reprocessing (after 
the v35r10p1 Brunel release) was completed within an hour yesterday.  The BK data taking conditions were updated 
by hand prior to launching the productions. In other news the 'Data Recovery Agent' has been running without 
intervention using the Moore and physics stripping as a test case these productions have achieved 100% of data 
processed (except for one file lost by RAL).  The final merging productions have been flushed.  The 134 error was 
reproduced at CERN and the problem was alleviated at some sites by unsetting the MALLOC_CHECK variable.  The 134 
error does persist at some sites (systematically) so the underlying problem should still be investigated.  Andrei agreed 
and commented that 25% of the current resources are wasted due to this problem and the other supposed bug in the 
finalisation of Gaudi that we are also trying to reproduce.  Vladimir reported that at some sites this reaches 100%.

The Castor migration time is causing a random delay for some files.  Andrew commented that the trigger is 300GB or 
8hrs pending but the Castors could tune the waiting time during this period.


2. DIRAC Release Update

- v4r21(p1,2,3) release
Minor bug fixes to dirac-architecture / GaudiApplication for MALLOC_CHECK. 


3. DIRAC Tasks 

- Framework status
Being tested on volhcb12.

- Bookkeeping status
Was updated and running on volhcb12. 

- WMS status
Being tested on volhcb12.

- DMS status
No reshuffling there was required. 

- Production Management status
Andrew commented that the production management system is reshuffled and LHCb specific components were 
refactored into LHCbDIRAC.  This is now running on volhcb12. 

- VO-boxes
No updates required.

- Web Portal
Being split into DIRAC and LHCb specific components.  The reintegration / deployment effort is ongoing and will be 
tested on volhcb12. 

- Services deployment ( hardware, configuration )
- Documentation

It was agreed to take the time to extensively test the pre-release in certification before Christmas and release the SVN 
based DIRAC in production. 

Roberto mentioned that the site status board was updated. A fine grained view of the production status is available 
there by job type.  SAM jobs are now linked with results. This will be linked from the DIRAC web portal.  The HC 
testing activity was discussed and if these results are to be exposed we should carefully discuss the failure cases.

Periodic infrastructure tests framework should be provided in DIRAC. ( ACTION: Stuart, Roberto )

4. AOB

Participation to the EGEE User Forum will be discussed offline once the most suitable tracks are identified.  It was 
suggested to have a DIRAC stand and this will be investigated.