19–25 Oct 2024
Europe/Zurich timezone

Building Scalable Analysis Infrastructure for ATLAS

23 Oct 2024, 17:09
18m
Large Hall B

Large Hall B

Talk Track 9 - Analysis facilities and interactive computing Parallel (Track 9)

Speaker

Lincoln Bryant (University of Chicago (US))

Description

We explore the adoption of cloud-native tools and principles to forge flexible and scalable infrastructures, aimed at supporting analysis frameworks being developed for the ATLAS experiment in the High Luminosity Large Hadron Collider (HL-LHC) era. The project culminated in the creation of a federated platform, integrating Kubernetes clusters from various providers such as Tier-2 centers, Tier-3 centers, and from the IRIS-HEP Scalable Systems Laboratory, a National Science Foundation project. A unified interface was provided to streamline the management and scaling of containerized applications. Enhanced system scalability was achieved through integration with analysis facilities, enabling spillover of Jupyter/Binder notebooks and Dask workers to Tier-2 resources. We investigated flexible deployment options for a "stretched" (over the wide area network) cluster pattern, including a centralized "lights out management" model, remote administration of Kubernetes services, and a fully autonomous site-managed cluster approach, to accommodate varied operational and security requirements. The platform demonstrated its efficacy in multi-cluster demonstrators for low-latency analyses and advanced workflows with tools such as Coffea, ServiceX, Uproot and Dask, and RDataFrame, illustrating its ability to support various processing frameworks. The project also resulted in a robust user training infrastructure for ATLAS software and computing on-boarding events.

Primary authors

David Jordan (University of Chicago (US)) Eric Christian Lancon (Brookhaven National Laboratory (US)) Farnaz Golnaraghi (University of Chicago (US)) Fengping Hu (University of Chicago (US)) Ilija Vukotic (University of Chicago (US)) Judith Lorraine Stephen (University of Chicago (US)) Lincoln Bryant (University of Chicago (US)) Robert William Gardner Jr (University of Chicago (US)) Ryan Paul Taylor (University of Victoria (CA))

Presentation materials