Automated recovery of data-intensive jobs in D0 and CDF using SAM

A. Baranovski1 (primary author)

V. Bartsch1,2 (presenter)

E. Lipeles3, A. Lyon1, I. Sfiligoi1,

D. Benjamin4, K. Genser1

1 Fermi National Accelerator Laboratory,

2University College London,

3University of California, 4Duke University

SAM
data handling system

allows distributed data handling for CDF, D0 and Minos

creates basis for easy GRID extensions (job handling, authentication, monitoring, brokering)

GRID extension diverse for CDF and D0

SAM

V. Bartsch, University College London

Data delivery

& caching

DH resource

management

SAM
data handling system

requirements:

scalability to transfer data

robust transfers with different types hardware over long distances

monitoring and book keeping

efficient use of remote resources

easy to handle system

=> solution for Fermilab experiments: SAM

V. Bartsch, University College London

SAM
distributed data handling

selected SAM stations

FNAL

DZero and CDF:

10k/20k Files declared/day

15k Files consumed/day

8 TByte of Files cons./day

main consumption of data still central

remote use on the rise

tests

start

V. Bartsch, University College London

CDF DAQ/Analysis Flow

Tevatron

Robotic Storage

with disk frontend

Reconstruction

Farm

CAF/DCAF

remote SAM

stations

local machines

at FNAL

data reduced

by triggers

via SAM

read & write

~3300 GHz CPU at FNAL

~2000 GHz CPU remote

V. Bartsch, University College London

Remote

MC production

(D)CAF: job handling

CAF UI

Start section

Worker Nodes

splits up job

End section

files

in storage

V. Bartsch, University College London

a typical analysis job consumes terabytes of information

during several days of running at a job execution site

Error cases: file delivery

CAF UI

Start section

Worker Nodes

splits up job

End section

files

in storage

V. Bartsch, University College London

Error cases: section restart

CAF UI

Start section

Worker Nodes

splits up job

End section

files

in storage

V. Bartsch, University College London

Error cases: output retrieval

CAF UI

Start section

Worker Nodes

splits up job into sections

End section

files

in storage

V. Bartsch, University College London

V. Bartsch, University College London

Error cases: statistics

CAF UI

Start section

Worker Nodes

End section

files

in storage

failure	causes

rates

3%	file retrieval failure

0.4%	segments restarts

job has 70 sections on average

3%	output retrieval failure

including user problems

status before SAM: analysis of the output files of the job

=> time consuming and depending on the output format

Error handling approach

Failure at any point of the CAF job:

=> time consuming parsing of the output files

=> whole file list, dataset, being reprocessed or lot of fixes by hand.

Approach:

job is monitored as a SAM project in the database

=> monitoring of file delivery errors

interface to experiment specific software (CAF) to decide on success of a job / sections

=> monitoring of section failures, output storage / certification failures, etc

Automatic recovery

CAF UI

Start section

Worker Nodes

splits up job,

starts services

for SAM

End section

stops services

for SAM,

keeps books

on sections

SAM database

keeps track of

each section

keeps track if

section finished

correctly

creates a recovery project with all those

information

V. Bartsch, University College London

status of the automatic recovery

file delivery problems are already caught

interfaces to the CAF built and currently under test

decision about the level of optimization not yet made

CAF deployment can be handled fast after the testing phase

Automatic recovery: conclusion

SAM takes care of the bookkeeping and allows to create recovery projects

interfaces between SAM and experiment specific software, e.g. CAF, can be built in order to incorporate the knowledge of the software into the recovery

=> error handling for the users much more transparent and effective

V. Bartsch, University College London

other SAM related contributions

no claim on completenes

Interface between the data handling system (SAM) and CDF experiment software

Experiences with operating SamGrid at the GermanGrid centre for CDF

The SAM-Grid / LCG interoperability system: a bridge between two Grids

Lightweight deployment of the SAM grid data handling system to new experiments

SAMGrid Peer-to-Peer Information Service

backup slide

D0 usage of SAM

typically 1PB/year are produced

transfer mechanisms used as tools

typical file sizes:

over 1GB for CDF

a few 100MB for Dzero

right now numbers higher for Dzero

because of reprocessing effort (GRID

based)

backup slide with Dzero numbers

Tevatron & Triggers (old data):

1.7MHz crossing rate

396ns clock cycle

Level 1: <40kHz accept rate

Level 2: <300Hz accept rate

Level 3: 75Hz accept rate

storage on the CAF: 180TB

storage on DCAFs: 78TB

local SAM stations:

Production Farm:

can process > 20 M events per day

(equivalent to ~3TB per day)

more than 400 CPU

Valeria Bartsch

2006-01-05T11:29:10

Valeria Bartsch