Computing in High Energy and Nuclear Physics (CHEP) 2012

The ATLAS LFC consolidation

22 May 2012, 13:30

4h 45m

Rosenthal Pavilion (10th floor) (Kimmel Center)

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session

Speaker

Fabrizio Furano (CERN)

Description

ATLAS decided to move from a globally distributed file catalogue to a central instance at CERN. This talk describes the ATLAS LFC merge exercise from the analysis phase over the prototyping and stress testing  to the final execution phase. We demonstrate that with careful preparation even major architectural changes could be implemented while minimizing the impact on the experiments production and analysis operations. Merging these large catalogues by processing partially inconsistent metadata for many tens of millions of files and replicas posed several challenges. We show how the new LFC instance was stress tested to ensure it met ATLAS's requirements and how the LFC schema evolved to support this. We also describe the main reasons why the merging process had to be done with a specialized multithreaded application. This was developed in order to accommodate the peculiarities that make this process much more challenging than a mere movement of data between SQL database instances. The process has to take into account a number of situations where the metadata records clash, or contain errors that have to be fixed on the fly, while still guaranteeing a high level of performance.