Resource control in ATLAS distributed data management: Rucio Accounting and Quotas

Apr 13, 2015, 3:30 PM
B250 (B250)



oral presentation Track4: Middleware, software development and tools, experiment frameworks, tools for distributed computing Track 4 Session


Martin Barisits (CERN)


The ATLAS Distributed Data Management system stores more than 160PB of physics data across more than 130 sites globally. Rucio, the next-generation data management system of ATLAS has been introduced to cope with the anticipated workload of the coming decade. The previous data management system DQ2 pursued a rather simplistic approach for resource management, but with the increased data volume and more dynamic handling of data workflows required by the experiment, a more elaborate approach to this issue is needed. This document describes how resources, like storage, accounts and replication requests, are accounted in Rucio. Especially the measurement of used logical storage space is fundamentally different in Rucio than it’s predecessor DQ2. We introduce a new concept of declaring quota policies (limits) for accounts in Rucio. This new quota concept is based on accounts and RSE (Rucio storage element) expressions, which allows the definition of account limits in a dynamic way. This concept enables the operators of the data management system to establish very specific limits in which users, physics groups and production systems use the distributed data management system while, at the same time, lowering the operational burden. This contribution describes the architecture behind those components, the interfaces to other internal and external components and will show the benefits made by this system.

Primary author


Cedric Serfon (CERN) Dr Mario Lassnig (CERN) Ralph Vigne (University of Vienna (AT)) Thomas Beermann (Bergische Universitaet Wuppertal (DE)) Vincent Garonne (CERN)

Presentation materials