Computing in High Energy and Nuclear Physics (CHEP) 2012

Development of noSQL data storage for the ATLAS PanDA Monitoring System

22 May 2012, 13:30

4h 45m

Rosenthal Pavilion (10th floor) (Kimmel Center)

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session

Speaker

Maxim Potekhin (Brookhaven National Laboratory (US))

Description

For several years the PanDA Workload Management System has been the basis for distributed production and analysis for the ATLAS experiment at the LHC. Since the start of data taking PanDA usage has ramped up steadily, typically exceeding 500k completed jobs/day by June 2011. The associated monitoring data volume has been rising as well, to levels that present a new set of challenges in the areas of database scalability and monitoring system performance and efficiency. These challenges have being met with a R&D and development effort aimed at implementing a scalable and efficient monitoring data storage based on a noSQL solution (Cassandra). We present the data design and indexing strategies for efficient queries, as well as our experience of operating a Cassandra cluster and interfacing it with a Web service.