14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

Processing of the WLCG monitoring data using NoSQL.

17 Oct 2013, 13:52
22m
Graanbeurszaal (Amsterdam, Beurs van Berlage)

Graanbeurszaal

Amsterdam, Beurs van Berlage

Oral presentation to parallel session Distributed Processing and Data Handling A: Infrastructure, Sites, and Virtualization Distributed Processing and Data Handling A: Infrastructure, Sites, and Virtualization

Speaker

Dr Edward Karavakis (CERN)

Description

The Worldwide LCG Computing Grid (WLCG) today includes more than 170 computing centres where more than 2 million jobs are being executed daily and petabytes of data are transferred between sites. Monitoring the computing activities of the LHC experiments, over such a huge heterogeneous infrastructure, is extremely demanding in terms of computation , performance and reliability. Furthermore, the generated monitoring flow is constantly increasing , which represents another challenge for the monitoring systems. While existing solutions are traditionally based on ORACLE for data storage and processing, recent developments evaluate NoSQL for processing large-scale monitoring datasets. NoSQL is an increasingly popular framework for processing datasets at the terabyte and petabyte scale using commodity hardware. In this contribution, we describe the integration of NoSQL data processing in the Experiment Dashboard framework and the first experience of using this technology for monitoring the LHC computing activities.

Primary author

Co-authors

Alexandre Beche (CERN) David Tuckett (CERN) Ivan Antoniev Dzhunov (University of Sofia) Ivan Kadochnikov (Joint Inst. for Nuclear Research (RU)) Jaroslava Schovancova (Acad. of Sciences of the Czech Rep. (CZ)) Julia Andreeva (CERN) Pablo Saiz (CERN) Sergey Belov (Joint Inst. for Nuclear Research (RU))

Presentation Materials