21-25 May 2012
New York City, NY, USA
US/Eastern timezone

The ATLAS LFC consolidation

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session

Speaker

Fabrizio Furano (CERN)

Description

ATLAS decided to move from a globally distributed file catalogue to a central instance at CERN. This talk describes the ATLAS LFC merge exercise from the analysis phase over the prototyping and stress testing  to the final execution phase. We demonstrate that with careful preparation even major architectural changes could be implemented while minimizing the impact on the experiments production and analysis operations. Merging these large catalogues by processing partially inconsistent metadata for many tens of millions of files and replicas posed several challenges. We show how the new LFC instance was stress tested to ensure it met ATLAS's requirements and how the LFC schema evolved to support this. We also describe the main reasons why the merging process had to be done with a specialized multithreaded application. This was developed in order to accommodate the peculiarities that make this process much more challenging than a mere movement of data between SQL database instances. The process has to take into account a number of situations where the metadata records clash, or contain errors that have to be fixed on the fly, while still guaranteeing a high level of performance.

Primary author

Co-authors

Cedric Serfon (Ludwig-Maximilians-Univ. Muenchen (DE)) Luca Canali (CERN) Marcin Blaszczyk (CERN) Simone Campana (CERN) Stewart Graeme Vincent Garonne (CERN)

Presentation Materials

There are no materials yet.