We upgraded Indico to version 3.0. The new search is now available as well.
10-15 March 2019
Steinmatte conference center
Europe/Zurich timezone

Continuous Analysis in ATLAS: Running User-Defined Container Images on the Grid

12 Mar 2019, 18:40
Steinmatte Plenary

Steinmatte Plenary

Oral Track 1: Computing Technology for Physics Research Track 1: Computing Technology for Physics Research


Lukas Alexander Heinrich (New York University (US))


The ATLAS software infrastructure has undergone several changes towards the adoption of Continuous Integration methodology to develop and test software. The users community can benefit from a CI environment in several ways: they can develop their custom analysis, build and test it using revision control services such as GitLab. By providing targeted official base images ATLAS enables users to also build self-contained Linux container images as part of the CI pipelines, a crucial component for analysis preservation and re-use scenarios such as reinterpretation of searches for Beyond the Standard Model physics (RECAST).

However, so far, the execution of preserved analyses was constrained to dedicated cloud infrastructure and not well-integrated into the wider WLCG computing model, where software distribution has so far relied on a combination of collaboration software distributed via the CVMFS filesystem and user software distributed ad-hoc by the workflow management system.

We describe an integration of containerized workloads into the grid infrastructure enabling users to submit self-authored or externally provided container images. To that end, the pilot process executed on the worker node has been extended to utilize the userspace container runtime singularity to execute such workloads. Further, the PanDA job configuration as well as the user-facing command line interfaces have been adapted to allow a detailed specification of the runtime environment.

Through this work a continuous grid analysis paradigm emerges, in which for each change in the revision control system, an automated pipeline of unit testing, image building and workload submission based on the freshly built image into the grid is triggered thus further streamlining physics analyses.

Primary authors

Lukas Alexander Heinrich (New York University (US)) Alessandra Forti (University of Manchester (GB)) Paul Nilsson (Brookhaven National Laboratory (US)) Tadashi Maeno (Brookhaven National Laboratory (US))

Presentation materials