Production and PASTE Meeting, 6th November 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: G.Castellani, S.Paterson, A.C.Smith
Phone: R.Bernet, N.Brook, G.Kuznetszov, U.Marconi, M.A.S.Miguelez, R.Nandakumar, A.Tsaregorodtsev
Apologies: M.Bargiotti, P.Charpentier, J.Closier, R.Santinelli
0. Ongoing Action List ( *New* )
Gathered from PASTE minutes of the previous two weeks.
23/10/07 - Marianne - Check dump of ProcessingDB in LFC and update
23/10/07 - Andrei - Add increased protection for user job deletion in DIRAC2
23/10/07 - Marianne - Request for basic documentation for restarting the Bookkeeping services
23/10/07 - Marianne - Bookkeeping should be migrated from SLC3 -> SLC4 and installed on volhcb05
23/10/07 - Ricardo, Andrei - Update Cleaning Agent to prevent user WMS problems
30/10/07 - Marianne - Check number of events for prod 1957 in BK
30/10/07 - Marianne - Update of LFC replica flags (C checked, N checked but not present)
30/10/07 - Roberto - Check LFC deployment status (request from last week, Roberto absent)
06/11/07 - Gianluca, Marianne - Remedy the Exceeded Max Staging Retry jobs and update ProcDB as required
06/11/07 - Gianluca, Marianne - Agents for automatic resolution of IntegrityDB entries should be developed
06/11/07 - Stuart - StagingSanityAgent should be developed and deployed
06/11/07 - Raja - Install v2r17p2 on VO-boxes
06/11/07 - Andrei - Contact ATLAS representative to understand VOMS usage in other VOs
06/11/07 - Andrei, Raja - Resolve issue with Raja's credential
06/11/07 - DIRAC developers - DIRAC3 (ongoing)
1. Production Status Report
Minutes from the production meeting are available
here.
Simulation is running without problems.
The recent Staging issues were raised by Gianluca. Stuart commented that changes to the StagerAgent have been
made to help with this. Now all staging requests are counted and therefore the state 'failed' 'exceeded max staging
retry' is final. A post-mortem Sanity check will be performed by a new component similar to the DataSanityAgent and
all information made available for the operations team such that a decision regarding input data can be made. (
ACTION: Stuart ) It was remarked that several Agents are now filling up the IntegrityDB but currently no actions are
subsequently taken. Andrei mentioned that the IntegrityDB should become primary tool for checking situation and
tools should be developed to act on the information in there. ( ACTION: Gianluca, Marianne )
Andrei also commented that when problematic files are discovered, these should be marked as 'problematic' in the
Processing DB and the remaining files 'unused'. This allows the recreation of such jobs. If all problematic datafiles can
be recovered, the affected job may simply be rescheduled. ( ACTION: Gianluca ) The Data Manager should be informed
of all irrecoverable datasets. ( ACTION: Marianne )
Gennady supplied dump of ProcessingDB to Marianne. This dump is still to be cross-checked in the LFC such that the
ProcDB can be updated. ( ACTION: Marianne )
2. Production Operations Issues
- v2r18 release
Andrei has released v2r17p2 as an intermediate testing release before v2r18. Stuart reported a couple of small
changes in the API, the StagerAgent modifications and the StagingSanityAgent that will become part of this release.
Andrei commented that the proxy informations can now be extracted via OpenSSL directly, some further debugging is
required for CERN certificates. Andrei remarked that a problem with Raja's credential had been discovered on the
VO-boxes and will be investigated offline. ( ACTION: Andrei, Raja ) Also on the VO-boxes, lcg-cp to Lyon is consistently
failing due to communication errors.
Nick suggested contacting an ATLAS representative to discover how they are using VOMS, a short presentation could
be made at the next PASTE meeting. ( ACTION: Andrei )
- LFC user names TCG discussion
Andrei reported that after discussion of the LFC 'nickname' (corresponding to the requested VOMS GA) the LFC
developers declined to implement this proposal. The decision about whether to maintain the pressure for this feature
or put the information in the CS and just manage everything ourselves should be taken.
There was a more positive response for the LFC admin interface. Jean-Philippe offered to supply certain administrative
scripts for our WMS instances that will become trusted hosts.
3. DIRAC3
Andrei conveyed that jobs can be submitted to the WMS and their status extracted. The optimizers can begin to be
ported. ( ACTION: Stuart ) Job Monitoring tools can also proceed since the JMS is running on volhcb03. ( ACTION:
Matvey ) Another major development is the Webpage configuration editing portal from Adri. This is secure with
configuration and authorization and this can now be extended for other services. This should now be installed on
volhcb03 to facilitate other developments. ( ACTION: Adri) Andrew has ported the Storage Elements of the DMS and
will eventually move onto the RM. Stuart has advanced with a Watchdog prototype and the full chain of job
submission to execution on the WN will now be defined. ( ACTION: Stuart ) Marcos has made some progress with the
Logging service.
A DIRAC3 distribution tool must be made available quickly in order to start testing clients.
There will be a brief DIRAC3 phone meeting on Thursday afternoon at 1530.
4. AOB
Andrei queried which CHEP papers have been submitted to the Journal for the conference but there seems to be no
way to get this information without asking the authors.
There are minutes attached to this event.
Show them.