We upgraded Indico to version 3.0. The new search is now available as well.
10-15 March 2019
Steinmatte conference center
Europe/Zurich timezone

Federated data storage evolution in HENP: data lakes and beyond

11 Mar 2019, 19:00
Steinmatte Plenary

Steinmatte Plenary

Oral Track 1: Computing Technology for Physics Research Track 1: Computing Technology for Physics Research


Andrey Zarochentsev (St Petersburg State University (RU))


Storage have been identified as the main challenge for the future distributed computing infrastructures: Particle Physics (HL-LHC, DUNE, Belle-II), Astrophysics and Cosmology (SKA, LSST). In particular, the High Luminosity LHC (HL-LHC) will begin operations in the year of 2026 with expected data volumes to increase by at least an order of magnitude as compared with the present systems. Extrapolating from existing trends in disk and tape pricing, and assuming flat infrastructure budgets, the implications for data handling for end-user analysis are significant. HENP experiments need to manage data across a variety of mediums based on the types of data and its uses: from tapes (cold storage) to disks and solid state drives (hot storage) to caches (including world wide access data in clouds and “data lakes”). DataLake R&D project aims at exploring an evolution of distributed storage while bearing in mind very high demands of HL-LHC era. Its primary objective is to optimize hardware usage and operational costs of a storage system deployed across distributed centers connected by fat networks and operated as a single service. Such storage would host a large fraction of the data and optimize the cost, eliminating inefficiencies due to fragmentation. In this talk we will highlight current status of the project, its achievements, interconnection with other research activities in this field like WLCG-DOMA and ATLAS-Google DataOcean, and future plans.

Primary authors

Xavier Espinal (CERN) Mr Andrey Kirianov (Petersburg Nuclear Physics Institut (RU)) Jaroslava Schovancova (CERN) Andrey Zarochentsev (St Petersburg State University (RU))

Presentation materials

Peer reviewing