Choose timezone
Your profile timezone:
PASTE meeting
→
Europe/Zurich
Description
CERN, bat 2 R-030
To join the telephone conference, click here: https://audioconf.cern.ch/call/0122882
To dial in to the conference:
a. Dial +41 22 7676000
b. Enter access code 0122882
Production and PASTE Meeting, 04th March 2008, CERN
Chair: A.Tsaregorodtsev
Minutes: S. Paterson
CERN: G.Castellani, J.Closier, Z.Mathe, S.Paterson, R.Santinelli
Phone: R.Bernet, N.Brook, R.Graciani, G.Kuznetszov, U.Marconi, M.Sapunov, A.C.Smith, A.Tsaregorodtsev
Apologies: M.Bargiotti, A.Casajus, P.Charpentier, M.A.S.Miguelez, R.Nandakumar, A.Zhelezov
0. Ongoing Action List
(01) - 06/11/07 - Gianluca - Remedy the Exceeded Max Staging Retry jobs and update ProcDB via development of Agents to resolve this
(04) - 13/11/07 - Marianne - RAL and PIC migrations, FTS job submissions
(07) - 27/11/07 - Zoltan, Elisa, Marianne - Prototype bookkeeping system on volhcb07
(09) - 04/12/07 - Roberto, Marcos, Gianluca - Develop a general service monitoring framework leading to a virtual control room
(25) - 08/01/08 - Marianne - Resolve why some files are double-counted in Bookkeeping
(27) - 08/01/08 - Andrew - Backup DMS services on lxgate34 to allow migration to SLC4
(36) - 12/02/08 - Roberto, Gianluca - Investigate other experiments usage and functionality of the Dashboard
(38) - 26/02/08 - Zoltan, Elisa, Marianne - Review the view-building process in the Bookkeeping
(39) - 26/02/08 - Joel - Test DIRAC v2r19 release before deployment
(41) - 26/02/08 - Marianne - Trigger replication of stripped data to other Tier-1s (RAL, IN2P3)
(42) - 04/03/08 - Marianne, Gianluca - Perform an integrity check on the files in the ProcessingDB
(43) - 04/03/08 - Joel - Submit pending strippping jobs after ProcDB integrity check
(44) - 04/03/08 - Roberto - Contact NIKHEF about increasing the length of available queues
(45) - 04/03/08 - Marianne - Add StoRm SE to DIRAC2 CS and replicate some DSTs there
(00) - 04/03/08 - DIRAC developers - DIRAC3 (ongoing)
Completed Actions
Done - (40) - 26/02/08 - Gianluca - Investigate causes of many aborted pilot agents by the LCG RBs
1. Production Status Report
Minutes from the production meeting are available here.
- Remedy Exceeded Max Staging Retry jobs and update ProcDB - Action(01)
Agent running on the test system, selects all data from IntegrityDB and performs checks on the file (lcg-ls, lcg-gt).
This is subsequently set to resolved if OK in the IntegrityDB and the status is updated in the ProcessingDB. The next
step is to move this into production after making the agent use the host certificate to avoid renewing proxies.
- RAL and PIC migrations - Action(04)
Everything containing PIC-tape has been removed from the catalog. For RAL after mounting two tapes at the same
time the transfers failed to work correctly, this issue is under investigation by Raja.
- Bookkeeping prototype - Action(07)
Zoltan reported that the Bookkeeping Manager is now running and work on porting the Agent to recreate the views is
starting. The genCatalog utility was used in DIRAC2 and this should be ported to DIRAC3 if required.
- Review the view-building process in the Bookkeeping - Action(38)
Elisa has finished the new schema and has tried to migrate the data to the new database. It was suggested to
discuss the view-building process next week.
- Test DIRAC v2r19 release before deployment - Action(39)
Pending verification from the Ganga team that the v2r18p11 release is OK.
- Investigate causes of many aborted pilot agents by the LCG RBs - Action(40)
Gianluca confirmed that the list-match is working as expected in the Agent Director but the situation changes after
submission and the RB Aborts these agents. It was agreed to close this action point.
- Trigger replication of stripped data to other Tier-1s - Action(41)
Andrei confirmed that the stripped data could also be sent to IN2P3.
Gianluca raised the issue of exceeding max staging retries on the production system, normally affecting a few files
per job. There was some discussion on how to react to the case when a tape is stuck at a site. The corresponding
integrity agent should be instrumented to warn the data manager of a problem in this case.
Recent processing jobs have been failing with 'Catalog Access' minor status. This is due to the inconsistent state of
the Processing DB. In DIRAC3 the LFC is used to populate the Processing DB so the issue is resolved. For now an
integrity check should be performed 'by hand' on the Processing DB. ( ACTION: Marianne, Gianluca ) Gianluca
retrieved all DIGI files from BK, ~45K files are still unused for the L0 stripping. It was agreed to continue with the
job submission after the integrity check. ( ACTION: Joel )
The issue of migrating data to StoRm at CNAF was raised by Umberto. Since the meeting it has been understood that
we should be able to run jobs reading from and uploading to StoRm with DIRAC2. Marianne has been asked to enter
the StoRm SE details in the DIRAC2 CS and replicate some stripped DSTs there. ( ACTION: Marianne )
2. CCRC08 Operations Issues
- Job execution status analysis per site
Ricardo mentioned data access issues are persisting at NIKHEF and IN2P3. Roberto mentioned that load on the
IN2P3 pnfs system (shared by all VOs) was causing many failures. The remainder of the sites are OK except for
intermittent file size determination problems that are under investigation.
- Heart Beat mechanism issues
Andrei updated the job state service to ensure that if the job is in the stalled state this is set back to the running
state. The load on the JobStateService should be understood, Andrei will send Ricardo some examples of failure due
to high load. Increasing the session length could improve the load on this service and will be investigated.
- Stalled jobs origin
Proxy expiration can be one cause of the stalled jobs, also CPU / WallClock limit being exceeded. It was agreed to
add a proxy renewal mechanism to the Watchdog.
- Configuration Service mirrors
A small number of jobs were failing to access the CS. The mirror at PIC will be set up for this.
- Proxy delivery issues
If VOMS servers fail to renew the requested proxy then the longest available is sent to the WN. Andrei suspects this
is the cause of intermittent failures to authenticate to the JobStateService.
3. DIRAC3 Tasks
There was some discussion about the workflows and it was agreed to aim for testing the full chain with Stripping
before May. The Stager should be developed in parallel for DIRAC3.
4. AOB
Roberto raised the issue of WallClock / CPU limits at the Tier-1s and agreed to contact NIKHEF about the length of
their queues. ( ACTION: Roberto )
There are minutes attached to this event.
Show them.