Speaker
Wen Guan
(University of Wisconsin (US))
Description
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service that has been deployed to serve data carousel, hyperparameter optimization, multiple-steps DAG (Directed Acyclic Graphs) workflows and so on. Here we will present the motivation for iDDS, the architecture, use cases and current status for ATLAS and Rubin Observatory exercise, and plans for the future.
Primary author
Wen Guan
(University of Wisconsin (US))
Co-authors
Aleksandr Alekseev
(Universidad Andres Bello (CL))
Brian Paul Bockelman
(University of Wisconsin Madison (US))
Fa-Hui Lin
(University of Texas at Arlington (US))
Tadashi Maeno
(Brookhaven National Laboratory (US))
Siarhei Padolski
(BNL)
Torre Wenaus
(Brookhaven National Laboratory (US))
Rui Zhang
(University of Wisconsin Madison (US))
