Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

An Information Aggregation and Analytics System for ATLAS Frontier

Nov 4, 2019, 2:00 PM
15m
Riverbank R8 (Adelaide Convention Centre)

Riverbank R8

Adelaide Convention Centre

Oral Track 4 – Data Organisation, Management and Access Track 4 – Data Organisation, Management and Access

Speaker

Andrea Formica (Université Paris-Saclay (FR))

Description

ATLAS event processing requires access to centralized database systems where information about calibrations, detector status and data-taking conditions are stored. This processing is done on more than 150 computing sites on a world-wide computing grid which are able to access the database using the squid-Frontier system. Some processing workflows have been found which overload the Frontier system due to the Conditions data model currently in use, specifically because some of the Conditions data requests have been found to have a low caching efficiency. The underlying cause is that non-identical requests as far as the caching are actually retrieving a much smaller number of unique payloads. While ATLAS is undertaking an adiabatic transition during LS2 and Run-3 from the current COOL Conditions data model to a new data model called CREST for Run 4, it is important to identify the problematic Conditions queries with low caching efficiency and work with the detector subsystems to improve the storage of such data within the current data model. For this purpose ATLAS put together an information aggregation and analytics system. The system is based on aggregated data from the squid-Frontier logs using the Elastic Search technology. This talk describes the components of this analytics system from the server based on Flask/Celery application to the user interface and how we use Spark SQL functionalities to filter data for making plots, storing the caching efficiency results into a PostgreSQL database and finally deploying the package via a Docker container.

Consider for promotion No

Primary authors

Nurcan Ozturk (University of Texas at Arlington (US)) Mrs Millissa Si Amer (Ecole Nationale Supérieure d'Informatique (DZ)) Andrea Formica (Université Paris-Saclay (FR)) Julio Lozano Bahilo (Univ. of Valencia and CSIC (ES)) Elizabeth Gallas (University of Oxford (GB)) Ilija Vukotic (University of Chicago (US))

Presentation materials