Speaker
Fabrizio Furano
(CERN)
Description
ATLAS decided to move from a globally distributed file catalogue to a central instance at CERN.
This talk describes the ATLAS LFC merge exercise from the analysis phase over the prototyping and stress testing to the final execution phase.
We demonstrate that with careful preparation even major architectural changes could be implemented while minimizing the impact on the experiments production and analysis operations.
Merging these large catalogues by processing partially inconsistent metadata for many tens of millions of files and replicas posed several challenges.
We show how the new LFC instance was stress tested to ensure it met ATLAS's requirements and how the LFC schema evolved to support this. We also describe the main reasons why the merging process had to be done with a specialized multithreaded application. This was developed in order to accommodate the peculiarities that make this process much more challenging than a mere movement of data between SQL database instances. The process has to take into account a number of situations where the metadata records clash, or contain errors that have to be fixed on the fly, while still guaranteeing a high level of performance.
Primary author
Fabrizio Furano
(CERN)
Co-authors
Cedric Serfon
(Ludwig-Maximilians-Univ. Muenchen (DE))
Luca Canali
(CERN)
Marcin Blaszczyk
(CERN)
Simone Campana
(CERN)
Stewart Graeme
Vincent Garonne
(CERN)
