The ATLAS Event Service: A new approach to event processing

13 Apr 2015, 17:15
15m
B250 (B250)

B250

B250

oral presentation Track4: Middleware, software development and tools, experiment frameworks, tools for distributed computing Track 4 Session

Speaker

Dr Torre Wenaus (Brookhaven National Laboratory (US))

Description

The ATLAS Event Service (ES) implements a new fine grained approach to HEP event processing, designed to be agile and efficient in exploiting transient, short-lived resources such as HPC hole-filling, spot market commercial clouds, and volunteer computing. Input and output control and data flows, bookkeeping, monitoring, and data storage are all managed at the event level in an implementation capable of supporting ATLAS-scale distributed processing throughputs (about 4M CPU-hours/day). Input data flows utilize remote data repositories with no data locality or pre­staging requirements, minimizing the use of costly storage in favor of strongly leveraging powerful networks. Object stores provide a highly scalable means of remotely storing the quasi-continuous, fine grained outputs that give ES based applications a very light data footprint on a processing resource, and ensure negligible losses should the resource suddenly vanish. We will describe the motivations for the ES system, its unique features and capabilities, its architecture and the highly scalable tools and technologies employed in its implementation, and its applications in ATLAS processing on HPCs, commercial cloud resources, volunteer computing, and grid resources.

Primary author

Dr Torre Wenaus (Brookhaven National Laboratory (US))

Co-authors

Kaushik De (University of Texas at Arlington (US)) Paolo Calafiura (Lawrence Berkeley National Lab. (US)) Paul Nilsson (Brookhaven National Laboratory (US)) Dr Peter Van Gemmeren (Argonne National Laboratory (US)) Tadashi Maeno (Brookhaven National Laboratory (US)) Vakho Tsulaia (Lawrence Berkeley National Lab. (US)) Wen Guan (University of Wisconsin (US))

Presentation Materials