14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

Rucio - The next generation of large scale distributed system for ATLAS Data Management

15 Oct 2013, 16:08
20m
Administratiezaal (Amsterdam, Beurs van Berlage)

Administratiezaal

Amsterdam, Beurs van Berlage

Oral presentation to parallel session Data Stores, Data Bases, and Storage Systems Data Stores, Data Bases, and Storage Systems

Speaker

Vincent Garonne (CERN)

Description

Rucio is the next-generation Distributed Data Management (DDM) system benefiting from recent advances in cloud and "Big Data" computing to address HEP experiments scaling requirements. Rucio is an evolution of the ATLAS DDM system Don Quijote 2 (DQ2), which has demonstrated very large scale data management capabilities with more than 140 petabytes spread worldwide across 130 sites, and accesses from 1,000 active users. However, DQ2 is reaching its limits in terms of scalability, requiring a large number of support staff to operate and being hard to extend with new technologies. Rucio will address these issues by relying on a conceptual data model and new technology to ensure system scalability, address new user requirements and employ new automation framework to reduce operational overheads. We present the key concepts of Rucio, including its data organization/representation and a model of how ATLAS central group and user activities will be managed. The Rucio design, and the technology it employs, is described, specifically looking at its RESTful architecture and the various software components it uses. We show also the performance of the system. We describe the strategy to roll-out the system during the first LHC long shutdown and how the transition from DQ2 to Rucio will be handled, including how Rucio will take advantage of the commissioning of new services, and how it will be integrated by external applications.

Primary author

Co-authors

Angelos Molfetas (University of Sydney (AU)) Armin Nairz (CERN) Cedric Serfon (CERN) Graeme Andrew Stewart (CERN) Dr Luc Goossens (CERN) Mario Lassnig (CERN) Martin Barisits (CERN) Ralph Vigne (University of Vienna (AT)) Thomas Beermann (Bergische Universitaet Wuppertal (DE))

Presentation Materials