Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

Metadata for fine-grained processing at ATLAS

Oct 13, 2016, 3:30 PM
1h 15m
San Francisco Marriott Marquis

San Francisco Marriott Marquis

Poster Track 4: Data Handling Posters B / Break

Speaker

Jack Cranshaw (Argonne National Laboratory (US))

Description

High energy physics experiments are implementing highly parallel solutions for event processing on resources that support
concurrency at multiple levels. These range from the inherent large-scale parallelism of HPC resources to the multiprocessing and
multithreading needed for effective use of multi-core and GPU-augmented nodes.

Such modes of processing, and the efficient opportunistic use of transiently-available resources, lead to finer-grained processing
of event data. Previously metadata systems were tailored to jobs that were atomic and processed large, well-defined units of data.
The new environment requires a more fine-grained approach to metadata handling, especially with regard to bookkeeping. For
opportunistic resources metadata propagation needs to work even if individual jobs are not finalized.

This contribution describes ATLAS solutions to this problem in the context of the multiprocessing framework currently in use for
LHC Run 2, development underway for the ATLAS multithreaded framework (AthenaMT) and the ATLAS EventService.

Primary Keyword (Mandatory) Data processing workflows and frameworks/pipelines
Secondary Keyword (Optional) Data model

Primary author

Jack Cranshaw (Argonne National Laboratory (US))

Presentation materials