Choose timezone
Your profile timezone:
PASTE meeting
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 23rd October 2007, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: M.Bargiotti, G.Castellani, J.Closier, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, U.Marconi, M.A.S.Miguelez, R.Nandakumar, A.Tsaregorodtsev
Apologies: P.Charpentier, A.C.Smith
1. Production Status Report
Minutes from the production meeting are available here.
Simulation, last remaining production sent yesterday, all requests from Olivier are complete.
Reconstruction, several issues: NIKHEF, GridKa, PIC and IN2P3, list of files which are physically present or not have
been sent to Marianne. The checking for these files is ongoing. srm-get-metadata was used by Marianne but Roberto
confirmed this does not tell you whether the file is physically present at the site. It was suggested to always submit a
GGUS ticket for such cases to discover from the site admins the physical status of problematic files. Andrei raised the
issue at the TCG and it was agreed the subject should get a larger forum, this is a longer term solution. The jobs
identified by the problematic LFNs should be identified and the causes of failure understood. Joel requested a list of
problematic files such that the ProcessingDB can be updated. ( ACTION: Marianne, Roberto )
The values for the LFC replica status flags should be synchronized with the service providers. Ongoing action from last
week to create a proposal for these characters and publish them on a Twiki page. ( ACTION: Marianne, Roberto )
Joel asked for information of replicas that are removed from the LFC. ProcDB setReplicaStatus() is not taken into
account for subsequent job reconstruction, Gennady should investigate. In the mean time only removal from the
Processing DB can resolve the situation.
Stuart mentioned that the problem of temporary (failover SE) replicas being added to the ProcessingDB creates a
mess for the data processing phase. There is also a problem of replicas that are in the processing DB and removed
from LFC and no automatic way to perform synchronization exists. Similarly, ancestor files could subsequently be
removed from the site where initial processing occurs, impacting on the stripping activity. Andrei agreed that for now,
the removal of replicas should be turned off in the Transfer Agents to minimize the damage. The longer term solution
of a failover LFC namespace with strict naming convention should also be pursued ( ACTION: Andrei )
Joel is currently testing the L0 stripping. The namespace DC06-v1r0 will be used for this. In Joel's absence over the
next fortnight Gianluca will chair the Monday and Thursday Operations meetings. The responsibility for SAM jobs also
passes to Gianluca. ( ACTION: Gianluca, Joel )
Joel mentioned that the shared area at Lyon is full, Andrei agreed to check the status with D.Bouvet. ( ACTION: Andrei )
2. Production Operations Issues
- v2r17 release
DIRAC v2r17 has been made available to Ganga for testing in the LHCBDEV area. Joel leaves for two weeks on Friday
and will make the release before then assuming no problems are discovered. Joel commented that a workflow module
was modified for the L0 stripping, this will be added to the v2r17 release.
- ProcessingDB clean-up
To summarize the discussion above, there will be no failover removal from this point on. Also there will be no
registration of failover replicas before the final destination SE to prevent accidental population of the ProcDB. A dump
of the ProcDB should be generated by Gennady and checked against the LFC by Marianne. Joel can then update the
ProcDB by hand to reflect the current status. ( ACTION: Gennady, Joel, Marianne )
- volhcb04,05 status
DIRAC2 installed on volhcb04 (production), and volhcb05 (user) but Andrei reported the host certificates were not
available. Joel is to check with Veronique about this. ( ACTION: Joel ) Bookkeeping will be migrated from SLC3 ->
SLC4 and installed on volhcb05. ( ACTION: Marianne )
- FTS2 tests and status
Roberto mentioned that the main changes for FTS2.0 were server side. Tests have been performed at IN2P3 and this
has been transparent. Moving to FTS2.0 can begin for LHCb.
- RAL Castor migration
Raja commented that RAL Castor is ready for migration. The data on dCache storage will be accessible during this
period. The scheduled downtime for data migration should be investigated. ( ACTION: Raja ) If the files can be
migrated in under 1 or 2 weeks, this may be more advantageous than establishing temporary SE names etc.
Marianne should investigate the estimated time for the LFC update since an intervention plan will be required. (
ACTION: Marianne )
- Job status change on the USER system
One user managed to delete other user jobs via job deletion over a range. Increased protection should be added for
this in DIRAC2. ( ACTION: Andrei )
Joel mentioned that one user managed to overload the software distribution server recently. After some discussion it
was agreed that a list of allowed software versions should be maintained such that JobSanityAgent can check this
requirement in the jobs and fail them meaningfully. Since this can still overload server, Hubert is working to store
software tarballs on a Grid SE. This will commence with one replica at a CERN SE before introducing replicas at other
SEs. ( ACTION: Hubert )
- Data cleanup status
Cleanup of SEs vs. LFC status was reported by Marianne. Physical namespace dump being checked against LFC. This
is ongoing. ( ACTION: Marianne )
- File access failure in user jobs
Stuart raised the recently reported problems with data integrity from users. Datasets that worked 2 weeks ago are
no longer functioning, DIGI and DST files affected. One file reported was from NIKHEF and has been lost, others are
seemingly OK. The genCatalog service could have been intermittently available to users when these problems were
reported. Joel requested basic documentation for restarting the Bookkeeping services. ( ACTION: Marianne )
3. VOMS GAs
The status of VOMS general attributes was given by Roberto. The message seems to be that EGEE projects will not be
VOMS GA aware. LHCb is interested in this for the user 'nicknames' and require VO admin tools for VOMSRS to allow
insertion of the VOMS GA's. These VOMS attributes are displayed via 'voms-proxy-info' commands etc. Roberto and
Joel will raise this at the VOMSRS meeting. ( ACTION: Roberto, Joel )
4. DIRAC3
Gennady, Alexey and Matvey are looking at the Production Management interface in Marseille. Andrew reported that
the transfer framework is almost complete. RequestDB, TransferDB, Replication Scheduler and FTSAgent are being
deployed on volhcb03. The intention is to begin transfers from CERN to participating Tier1s for SRM2 testing by the
end of the day. Stuart reported that work has started on the DIRAC3 Watchdog in preparation for the DIRAC3 Job
Wrapper.
5. AOB
Nick raised a question from PIC regarding LFC service. Should they wait for the latest LFC RPMs to become available?
The timescale for certification is expected to be one month. Roberto agreed to confirm the timing of the availability of
this release and notify Gonzalo about this. ( ACTION: Roberto )
Joel queried the status of the Cleaning Agent on Production, Test and User systems. Andrei did not have time to look
at this and Ricardo is currently unavailable. ( ACTION: Andrei )
There are minutes attached to this event.
Show them.