Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 9th October 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, J.Closier, S.Paterson, R.Santinelli
Phone: G.Kuznetszov, U.Marconi, M.A.S.Miguelez, R.Nandakumar, M.Sapunov, A.Tsaregorodtsev, A.Zhelezov
Apologies: P.Charpentier, R.Bernet, N.Brook
1. Production Status Report
Minutes from the production meeting are available here.
No simulation running (pending v2r16 release). Tests not completely successful due to unsuccessful writing to the
LFC. v2r16 now deployed on production system, jobs run successfully after the reboot of the LFC host. PIC, many
job failures, must check whether problem is data related or due to the lack of tape drives there. Staging at GridKa has
still been problematic, the issue was raised at the Operations meeting. Site responsibles are investigating a possible
bug in dCache as well as the site SRM configuration.
Stuart raised the point that there is almost nothing we can do on our side to improve the situation, all other Tier-1
sites are able to handle the load. Alexey reported local staging times of ~1-2hrs and this is not observed via LCG
utilities and SRM. This should be investigated. ( ACTION: Stuart, Joel, Alexey )
The reason for no running jobs on the production system yesterday (out of 120) was due to '-SLC4' sites. Jobs
pending for RAL are waiting due to list-match showing zero available CEs, Roberto agreed to investigate whether there
is a BDII related problem. ( ACTION: Roberto )
At Lyon there are several jobs with 'Exceeded max staging retry', Marianne should confirm the status of these files to
allow the recreation of these jobs in the Processing DB. ( ACTION: Marianne )
Upgrade of Castor at CERN: Marianne will provide the list of replicas on srm-durable at CERN. The Castor people can
then create a new pool and copy all the files there. ( ACTION: Marianne )
Philippe circulated a plan for storage migrations e.g. PIC / RAL. The decision on the migration can be taken after the
ongoing cleanup. We shall continue to support DC04 data at Castor sites but can physically remove replicas from the
dCache sites now. ( ACTION: Marianne )
2. Production Operations Issues
- v2r16 release status
v2r16 has been deployed in the production system.
- LFC updates
The recent reboot of the LFC hosts has resolved the race condition problems. Roberto confirmed that the LFC update
with the flag for disabling automatic registrations of new DN is now ready and tested. This will be released with the
next version of the LFC server. Replication of the administrative tables should be introduced to CNAF prior to Lyon
and RAL. Joel has requested an LFC admin interface allowing manipulation of group / role (ticket open > 1 week).
Roberto agreed to chase this up. ( ACTION: Roberto )
- SE-LFC integrity checking status
NIKHEF acknowledge the coupling between FC and SE is not tight enough. The response was that providing the
dump of the namespace of files on the physical storage is not scalable and the issue will be raised at the TCG. Alexey
agreed to perform this check at GridKa. ( ACTION: Alexey ) PNFs has been mounted on the VO-box at Lyon such that
standard commmands can be used there. ( ACTION: Andrei ) Roberto will perform the operation at CNAF. ( ACTION:
Roberto ) Andrew has developed an agent to use local backend commands to query the physical storage from the
VO-boxes. Raja should install this agent at Lyon to test everything is working. ( ACTION: Raja ) Running the agent
without setting the LFC flag to mark a replica as checked is not worthwhile, Marianne will investigate this
functionality. ( ACTION: Marianne )
- Checking before data usage in ProcDB
Gennady has implemented a check on the LFC flag such that jobs are constructed only on 'OK' files. Without an
on-demand agent performing checks, this doesn't make sense and could cause large delays in the full chain. It was
agreed to continue with the current mechanism for now. Stuart reported that the Data Sanity Agent is almost ready,
the ProcessingDB and IntegrityDB can be notified of problematic files such that the process of recreating jobs will be
greatly simplified. ( ACTION: Stuart )
- DIRAC.CERN.ch status
No progress as of yet, Andrei and Stuart agreed to make an effort to deploy this during the week. ( ACTION: Andrei,
Stuart ) Joel is still to investigate the status of LSF clients on the WMS machines and whether the lxbatch queues are
suitable. If no additional resources are gained for LHCb, this may not be worthwhile, Joel agreed to clarify the
situation. ( ACTION: Joel )
- New volhcbXX hosts status
Joel confirmed that the machines will be ready in 1 week.
- Application of the default VOMS schema for user mapping
The default VOMS schema roles have been defined. Joel confirmed that switching from group to role, the same
actions can be performed. Ongoing problems are being discovered with site VOMS mappings (where the default role
mappings are not in place). This will be debugged using the DIRAC SAM jobs. Roberto agreed to help to interface with
the problematic sites and update the YAIM configuration to reflect LHCb needs. Current FQAN mappings in LFC should
also be understood. For dCache sites, how gPlasma configuration affects the ownership etc. ( ACTION: Joel, Roberto )
The VOMS proxy renewal mechanism should be introduced in parallel with the gLite WMS. Stuart suggested the
lhcb-proxy-init script should be established in order to ensure user credentials are uploaded to the MyProxyServer.
- Inconsistency of cached status of files at CNAF
Some user jobs had consistently failed to access data on the WN whilst SRM reported an OK status. Jobs would run at
CNAF until the wall-clock time expired. Recovering from this should be done with error reporting from the DIRAC3 Job
Wrapper.
3. DIRAC3 Tasks
Logging service updates completed by Marcos, the changes will be committed by the end of the day. Andrew has
been working on the generic requests such that the DM specific part can be removed and made into a module.
Gennady has been working on the production management database but has problems obtaining information from
the CS, this is ongoing.
4. AOB
Raja raised the issue of RAL-Castor migration. As discussed earlier for PIC, the sanity of data at RAL should be
checked in advance. Andrew agreed to send Raja comments regarding their proposal for SRM endpoint configuration.
( ACTION: Raja)
There are minutes attached to this event.
Show them.