Speaker
Mario Lassnig
(CERN)
Description
Rucio is the successor of the current Don Quijote 2 (DQ2) system for the distributed data management (DDM) system of the ATLAS experiment. The reasons for replacing DQ2 are manifold, but besides high maintenance costs and architectural limitations, scalability concerns are on top of the list.
The data collected so far by the experiment adds up to about 115 Peta bytes spread over 270 million distinct files. Current expectations are that the amount of data will be three to four times as it is today by the end of 2014. Further is the expansion of the WLCG computing resources pushing additional pressure on the DDM system by adding more powerful computing resources, which subsequently increases the demands on data provisioning. Although DQ2 is capable of handling the current workload, it is already at its limits. To ensure that Rucio will be up to the expected quality of service, a way to emulate the expected workload is needed. To do so, first the current workload observed in DQ2 must be understood in order to scale it up to future expectations.
This paper presents an overview of the theory behind workload emulation and discusses how selected core concepts are applied to the workload of the experiment. Further a detailed discussion is provided on how knowledge about the current workload is derived from central file catalogue logs, PanDA dashboard, etc. The discussion also addresses how this knowledge is utilized in the context of the emulation framework. Finally a description of the implemented emulation framework used for stress-testing Rucio is given.
Primary author
Ralph Vigne
(University of Vienna (AT))
Co-authors
Armin Nairz
(CERN)
Cedric Serfon
(CERN)
Erich Schikuta
(University of Vienna)
Graeme Andrew Stewart
(CERN)
Dr
Luc Goossens
(CERN)
Mario Lassnig
(CERN)
Martin Barisits
(CERN)
Thomas Beermann
(Bergische Universitaet Wuppertal (DE))
Vincent Garonne
(CERN)
