Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

Production Experience with the ATLAS Event Service

Oct 11, 2016, 3:00 PM
15m
GG C3 (San Francisco Mariott Marquis)

GG C3

San Francisco Mariott Marquis

Oral Track 4: Data Handling Track 4: Data Handling

Speaker

Vakho Tsulaia (Lawrence Berkeley National Lab. (US))

Description

The ATLAS Event Service (ES) has been designed and implemented for efficient
running of ATLAS production workflows on a variety of computing platforms, ranging
from conventional Grid sites to opportunistic, often short-lived resources, such
as spot market commercial clouds, supercomputers and volunteer computing.
The Event Service architecture allows real time delivery of fine grained workloads to
running payload applications which process dispatched events or event ranges
and immediately stream the outputs to highly scalable Object Stores. Thanks to its agile
and flexible architecture the ES is currently being used by grid sites for assigning low
priority workloads to otherwise idle computing resources; similarly harvesting HPC resources
in an efficient back-fill mode; and massively scaling out to the 50-100k concurrent core
level on the Amazon spot market to efficiently utilize those transient resources for peak
production needs. Platform ports in development include ATLAS@Home (BOINC) and the
Goggle Compute Engine, and a growing number of HPC platforms.

After briefly reviewing the concept and the architecture of the Event Service, we will
report the status and experience gained in ES commissioning and production operations
on various computing platforms, and our plans for extending ES application beyond Geant4
simulation to other workflows, such as reconstruction and data analysis.

Primary Keyword (Mandatory) Data processing workflows and frameworks/pipelines
Tertiary Keyword (Optional) Distributed data handling
Secondary Keyword (Optional) Distributed workload management

Primary authors

Doug Benjamin (Duke University (US)) Kaushik De (University of Texas at Arlington (US)) Paolo Calafiura (Lawrence Berkeley National Lab. (US)) Paul Nilsson (Brookhaven National Laboratory (US)) Peter Van Gemmeren (Argonne National Laboratory (US)) Tadashi Maeno (Brookhaven National Laboratory (US)) Taylor Childers (Argonne National Laboratory (US)) Torre Wenaus (Brookhaven National Laboratory (US)) Vakho Tsulaia (Lawrence Berkeley National Lab. (US)) Wen Guan (University of Wisconsin (US))

Presentation materials