10–14 Jan 2022
Online only
Europe/London timezone

An intelligent Data Delivery Service (iDDS) for and beyond the ATLAS experiment

10 Jan 2022, 16:24
1m
Online only

Online only

Poster R&D R&D

Speaker

Wen Guan (University of Wisconsin (US))

Description

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service that has been deployed to serve data carousel, hyperparameter optimization, multiple-steps DAG (Directed Acyclic Graphs) workflows and so on. Here we will present the motivation for iDDS, the architecture, use cases and current status for ATLAS and Rubin Observatory exercise, and plans for the future.

Primary author

Wen Guan (University of Wisconsin (US))

Co-authors

Aleksandr Alekseev (Universidad Andres Bello (CL)) Brian Paul Bockelman (University of Wisconsin Madison (US)) Fa-Hui Lin (University of Texas at Arlington (US)) Tadashi Maeno (Brookhaven National Laboratory (US)) Siarhei Padolski (BNL) Torre Wenaus (Brookhaven National Laboratory (US)) Rui Zhang (University of Wisconsin Madison (US))

Presentation materials