Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 11th March 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: G.Castellani, J.Closier, R.Graciani, G.Kuznetszov, Z.Mathe, M.A.S.Miguelez, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, M.Sapunov, A.C.Smith, A.Tsaregorodtsev
Apologies: M.Bargiotti, A.Casajus, P.Charpentier, R.Nandakumar, A.Zhelezov
0. Ongoing Action List
(01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(25) - 08/01/08 - Zoltan - Resolve why some files are double-counted in Bookkeeping
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(36) - 12/02/08 - Roberto, Gianluca - Investigate other experiments usage and functionality of the Dashboard
(38) - 26/02/08 - Zoltan, Elisa, Marianne - Review the view-building process in the Bookkeeping
(41) - 26/02/08 - Marianne - Trigger replication of stripped data to other Tier-1s (RAL, IN2P3)
(42) - 04/03/08 - Marianne, Gianluca, Joel - Perform an integrity check on the ProcDB and submit pending jobs
(44) - 04/03/08 - Roberto - Contact NIKHEF about increasing the length of available queues
(46) - 11/03/08 - Angelo, Roberto - Send some test jobs to CNAF to test StoRM
(00) - 11/03/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (45) - 04/03/08 - Marianne - Add StoRm SE to DIRAC2 CS and replicate some DSTs there
Done - (39) - 26/02/08 - Joel - Test DIRAC v2r19 release before deployment
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Gianluca had problems in enabling the agent use the host certificate so is running this as an interactive script for the
time-being. Basic checks for the disk replicas are in place but further checks to confirm the GUIDs in the BK are
pending. The timescale for finalizing this is next week.
- RAL and PIC migrations - Action(04)
There are ~56 remaining tapes at RAL corresponding to ~6 weeks before completion (in the most optimistic case).
Andrew suggested replication of all possible data from CERN to speed up the process. Andrei requested a mail be
sent to the developers list to outline this procedure. Gianluca commented that the list provided to Marianne from PIC
was short by ~7K files (Joel confirmed these are test files / .digi files that can be ignored).
- Bookkeeping prototype - Action(07)
Elisa hasn't yet migrated the existing BK data to the new prototype instance of Oracle with updated schema. Zoltan
reported that the Bookkeeping Manager Agent is ported to DIRAC3. This is now running smoothly after final testing
last week. The Agent performing the recreation of the views is under development pending the migration of the data
by Elisa.
- Monitoring framework - Action(09)
Roberto put in place some SAM sensors for the LFC last week and noticed that some instances are not correctly
publishing information into the BDII.
- Double-counted files in the BK - Action(25)
Zoltan has isolated the problem and is going to fix this.
- lxgate34 Status - Action(27)
This machine can be retired on a similar timescale to DIRAC2.
- Investigate other experiments usage of the Dashboard - Action(36)
To be deferred to the discussion next week with James and Julia.
- Review the view-building process in the Bookkeeping - Action(38)
The Agent performing the recreation of the views is under development pending the migration of the data by Elisa.
- Test DIRAC v2r19 release before deployment - Action(39)
The Ganga team confirmed that the v2r18p11 (v2r19) release is OK and this is installed at CERN.
- Trigger replication of stripped data to other Tier-1s - Action(41)
This should proceed urgently to provide more sites for ongoing user tasks.
- Integrity check on the files in the ProcessingDB - Action(42)
Gianluca confirmed that there are still problems remaining since the mis-manipulation of the ProcDB. The data
integrity check against the LFC is still pending. This action is to be merged with Action(43) and
Gianluca is asked to confirm with Joel that jobs have been submitted for the fixed entries.
- Queue length at NIKHEF - Action(44)
Roberto confirmed that a GGUS ticket is still open about this issue.
- Add StoRm SE to DIRAC2 CS and replicate some DSTs there - Action(45)
Andrew confirmed this had been done yesterday. Nick suggested Angelo could formulate the test job and submit
these to CNAF. ( ACTION: Angelo, Roberto )
Joel reported that disk to disk copies at CERN have recently been failing and has opened a GGUS ticket for the issue.
Gianluca reported that Marianne found ~1K replicas registered with the port number and will trigger the cleanup of
this. Jobs were rescheduled with 'Input Data Not Available' after a typo was discovered in the SiteLocalSEMapping for
RAL but since the optimizer was not restarted it was assumed that this is the cause. Marcos had some proxy related
problems and it was agreed to debug this offline. It was agreed to go through the site and SE masks each week to
keep these up to date.
2. CCRC08 Operations Issues
- pyGSI deployment
The framework was updated yesterday to use pyGSI instead of pyOpenSSL and all central services were restarted. A
new python tarball has to be constructed from CVS.
- Tagged DIRAC3 beta release
The procedure for tagging will be discussed via email.
- volhcb01 DIRAC3 installation
Joel observed a memory leak problem on volhcb01. Ricardo believes that this is due to some services still using
pyOpenSSL instead of pyGSI and will confirm this.
- CS master on volhcb01 + mirrors
Mirrors for the CS are becoming urgent and Andrei suggested making a CS Master instance on volhcb01. Joel
commented that a problem was observed retrieving the list of servers for the CS.
Stuart commented that if hardcoding the CS URLs can be avoided by passing this as an argument to the pilot it may
save future troubles.
3. CCRC results and further DIRAC3 tasks
The list of pending CCRC phase 2 tasks compiled by Nick and Andrei is copied below with brief notes:
1. Finalise the error catching of the current CCRC08 reconstruction workflow
- Full debugging information is required after a review of the phase 1 errors
- Continue submitting jobs to the Grid with DIRAC3 to catch all errors
2. Prepare the stripping workflow, prepare the MC workflow
- Joel is continuing this work with Gennady
- The resolution of ancestor files is pending a DIRAC3 version of genCatalog
- Workflow editors will be discussed next week
3. Resolve the file size issue with regards the file upload
- Investigations are ongoing, Andy
4. DIRAC3 staging in place
- Can be performed next week
5. DIRAC3 failover in place
- Deploying new request service on all the VO-boxes
6. Review what is available in the monitoring in light of February's activity,
this should feed into the DIRAC3 week (review liaison with LCG activity such
as Dashboards)
- Fix time with James / Julia (proposal is for next Thursday afternoon)
7. Understand the use/reqts of the Gaudi application using the conditions DB
in the Grid DB (In particular closest location of DB)
- Timescale for tests is this week
8. Review timescales of glexec & testing and is it consistent with phase 2 1st May
- Andrei confirmed that glexec may not be ready for phase 2
9. Use of a "time left" utility to allow multiple payloads for a pilot agent
and signalling applications
- Stuart reported that the development of this is ongoing and progress is being made
- Roberto is triggering discussions with the Tier-1 sites about the advertised limits
- Will put a standard unit in place throughout DIRAC and consider best benchmarking approach
10. File merging for MC - low priority as MC data is on TxD1 storage so small
file size shouldn't be a big issue
11. Accounting information storage and visualization
- Ricardo and Adri are working on this
12. User level documentation for production and analysis activities.
- Effort is required here before release but without compromising CCRC phase 2 activity
4. AOB
There was some discussion of the SW week talks and the agenda was agreed. Roberto confirmed that the LFC
instance at GridKa is now available. Andrei asked whether there was a procedure to check when a mirror was last
updated and Roberto agreed to look into this. The PIC instance is envisaged for mid-April.
There are minutes attached to this event.
Show them.