Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 16th October 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, P.Charpentier, J.Closier, G.Kuznetszov, S.Paterson, R.Santinelli
Phone: N.Brook, A.Carbone, U.Marconi, M.A.S.Miguelez, A.Tsaregorodtsev
Apologies: R.Bernet
1. Production Status Report
Minutes from the production meeting are available here.
Simulation has produced some data but one workflow had a missing GENERATOR_NAME. M.Cattaneo reported many
emails from GridKa for applications at various points of failure. The same production runs fine elsewhere so most
likely this is an NFS related issue. Joel confirmed that the strange cmt/ directory problem reported by Alexey is not
related to this.
Issues with Reconstruction at GridKa have been resolved and jobs are again running there. Alexey reported that
some data is still not accessible and this should be pursued. (ACTION: Alexey )
CNAF, problems with the Pilot Agent accessing the Configuration Service and failing. The cause of this was a
corruption in the site shared area which has been understood by Angelo and fixed. Tests to confirm the resolution of
this are pending before unbanning CNAF. ( ACTION: Joel ) Stuart suggested introducing a simple test to retrieve
arbitrary information from the CS in order to establish that there are no problems with the site. If this doesn't work
then problems can immediately be reported to std.out std.err in DIRAC2 (and the Logging serivce in DIRAC3).
A GGUS ticket at SARA asking for the status on corrupted datafiles was recently marked as 'solved' without
explanation, subsequently it was discovered that most of these files were lost.
Existing physical files on dCache have not been removed from storage but are removed from the LFC. Trying to
replicate to the same SE would result in failure (file exists on PNFS). Without changing the SURL there doesn't seem
to be a way to force the re-replication. It was agreed to open a GGUS ticket per request since no other tools are
available for this.
Angelo connected and mentioned that the problem at CNAF was related to the migration of the shared areas to GPFS.
After some discussion it was agreed that any important changes in site configuration should be broadcasted at a high
level. Roberto agreed to raise the issue at the Operations meeting.
The Data Sanity Agent is now reporting for 'stalled' and 'failed', 'max reschedules' jobs to the Integrity DB (> 200
problems found to date) and it was mentioned that the Stager Agent should now be modified to report both to the
IntegrityDB and the ProcessingDB as well. ( ACTION: Stuart )
2. Production Operations Issues
- User system disk shortage problem
Input / output sandboxes of the users have grown very fast. A combination of non-efficient cleaning and the sharing
of a fairly small /opt/ partition on the WMS machine with Bookkeeping was causing problems. Andrei migrated the
output sandbox table to the 1TB storage area. The Cleaning Agent should be updated and the timing of data
retention should be announced to the users. ( ACTION: Andrei )
- Migration to new volhcbXX hosts
Ports need to be opened for the new machines and Joel was asked to fill out a form for each port to describe each
service and protocol used. Andrei reported that the users on the new machines were not properly defined. A date
should be defined for the migration when the groups are correct. ( ACTION: Joel )
- Using VOMS proxies for production
The use of VOMS proxies in production was reported by Joel. For all the sites, the SGM account appears to be
correctly mapped. Roberto confirmed the same for the production accounts (after submission of several GGUS
tickets). The switch to VOMS proxies must go in parallel with using the gliteRBs. Stuart queried whether the gLite
middleware version was consistent across the CERN and CNAF RBs and Roberto reported this was the case.
- LFC update
Roberto gave a summary of the LFC status. Flag to disable automatic registration of new users should be available
soon. Andrei will report the request for LFC admin interface as well as the addition of VOMS attribute information in
the LFC at the TCG tomorrow. ( ACTION: Andrei ) The notion of secondary groups in the LFC should be removed for
LHCb. Roberto agreed to report this to SA3 and follow up with the deployment team such that LFC instances for LHCb
will not be equipped with the middleware software supporting secondary groups. ( ACTION: Roberto ) Setting group
writable directories for all LHCb users will also be pursued as a temporary solution. ( ACTION: Joel, Roberto )
- DIRAC.CERN.ch
The question of whether additional share is gained from an Agent sitting on LXBATCH rolls over from last week.
Philippe clarified that all other VOs are doing this and to offload Grid workload management services we can go directly
to LSF submission via a dedicated DIRAC.CERN.ch Agent. Job Agent should be modified to download the available
proxy if the owner of the job is different. This will be undertaken after migration to the volhcbXX hosts. ( ACTION:
Andrei , Stuart )
- Data cleanup status
Marianne mentioned that DC04 PIC-tape replicas should be transferred to CERN, removal is now starting. The PIC
dump for Adri can now be performed. ( ACTION: Marianne ) Andrei suggested the following simple file states: Replica
unchecked; Replica checked and Replica bad. A proposal for defining the specific tests for marking a replica as OK
should be defined. ( ACTION: Marianne )
Nick mentioned that the dress-rehearsal activity may require additional planning. Philippe suggested an ad-hoc
meeting during the next DIRAC3 developers week (19th November). ( ACTION: Nick )
3. DIRAC3
Tasks are ongoing. It was agreed to postpone the discussion until more developers are present.
4. AOB
None
There are minutes attached to this event.
Show them.