Production and PASTE Meeting, 30th October 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, S.Paterson, A.C.Smith
Phone: R.Bernet, N.Brook, G.Kuznetszov, M.A.S.Miguelez, A.Tsaregorodtsev
Apologies: J.Closier, R.Santinelli
1. Production Status Report
- Report from the Production meeting
Gianluca reported that production 1959 has been submitted. 800 of the 1K jobs are still in the staging status with 2
jobs completed so far. Recent 'proxy expired' status was due to Raja's credential expiring in the myproxy server
before job submission. Matvey and Stuart also observed some inconsistent behaviour with the myproxy server. The
status of proxies in the repository could not be queried and seemed to indicate that no proxies were present. There
was some discussion of how to report this problem in Roberto's absence. Stuart suggested submitting a GGUS ticket
after some more tests. Andrei agreed to communicate this to Matvey. ( ACTION: Andrei, Matvey )
Productions: 1960, 1962 and 1963 were submitted by Marcos, there are many Aborted pilots associated with this.
These pilots are aborted at the site level so the sites inside the mask should be carefully checked. ( ACTION: Gianluca )
Problematic files found from staging failures as reported by the Stager, Marianne reported these files to Castor support
and is awaiting a response.
Stuart wondered whether any minutes from the Monday and Thursday meetings were available. Gianluca agreed to
compile these and add them to the production in progress Twiki page
here. ( ACTION: Gianluca )
For production 1957 Marcos commented that only 2500 events have been registered in Bookkeeping. Further
investigations showed 35K events have actually been produced. This should be checked. ( ACTION: Marianne )
- ProcessingDB clean-up
As discussed last week, the Processing DB is in an inconsistent state due to replicas not existing after failover cleanup
operations. The dump of files from the Processing DB should be passed to Marianne for checking. ( ACTION: Gennady )
2. Production Operations Issues
- Production System on volhcb04 and v2r17p1 on VO-boxes
Andrei reported that the migration has been fairly smooth and v2r17 is running on volhcb04. There were some
problems setting up the correct environment for VOMS usage everywhere. Matvey, Andrei and Stuart debugged this
on Friday afternoon. The same problem has been identified on the VO-boxes where either VOMS proxy commands are
OK and the LFC interaction is affected and vice-versa. Investigations for this are ongoing, however, this is a
show-stopper since VOMS proxy commands seem to depend on the local environment in a strange way. Andrei
agreed to contact the VOMS developers. ( ACTION: Andrei ) Nick also suggested to contact other experiments who
have already migrated to VOMS.
- RAL storage migration
Philippe remarked that for the hundreds of TB existing on Castor there seems to be no way to assign a space token for
the SRM2 migration. This issue should be raised at the GSSD. ( ACTION: Nick ) After some discussion, Philippe
suggested an initial migration from dCache to Castor that can proceed with proper SE name, coordinated by Marianne
and Raja. Two endpoints can be used with a single SA path, corresponding to new RAL-tape d1-t1 and RAL-disk d1-t0.
Andrei suggested to make an iteration with the RAL site managers to suggest this approach, the intervention plan
should be modified to reflect this and passed to Raja. ( ACTION: Nick, Raja )
- LFC status
The status of the LFC deployment was queried by Andrei but in Roberto's absence this will roll-over to next week.
Marianne update of flags at RAL (C checked, N checked but not present) and the ongoing process is starting at Lyon. (
ACTION: Marianne )
3. DIRAC3
Stuart suggested a scheduled DIRAC3 meeting on Thursday afternoon, details to be confirmed. Andrei agreed to
make a proposal to the DIRAC3 development list. This Thursday the meeting will be 1530.
4. AOB
Nick raised the issue of response times for critical service failure. Within 2hrs in the saturated regime, e.g. 10K
concurrent jobs, 10% of the jobs will complete. Using 2hrs as a benchmark for critical service downtime seems to be
sensible.
There are minutes attached to this event.
Show them.