Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

The ATLAS EventIndex General Dataflow and Monitoring Infrastructure

Oct 11, 2016, 2:00 PM
15m
GG C3 (San Francisco Mariott Marquis)

GG C3

San Francisco Mariott Marquis

Oral Track 4: Data Handling Track 4: Data Handling

Speaker

Alvaro Fernandez Casani (Instituto de Fisica Corpuscular (ES))

Description

The ATLAS EventIndex has been running in production since mid-2015,
reliably collecting information worldwide about all produced events and storing
them in a central Hadoop infrastructure at CERN. A subset of this information
is copied to an Oracle relational database for fast access.
The system design and its optimization is serving event picking from requests of
a few events up to scales of tens of thousand of events, and in addition, data
consistency checks are performed for large production campaigns. Detecting
duplicate events with a scope of physics collections has recently arisen as an
important use case.
This paper describes the general architecture of the project and the data flow
and operation issues, which are addressed by recent developments to improve the
throughput of the overall system. In this direction, the data collection system
is reducing the usage of the messaging infrastructure to overcome the
performance shortcomings detected during production peaks; an object storage
approach is instead used to convey the event index information, and messages to
signal their location and status. Recent changes in the Producer/Consumer
architecture are also presented in detail, as well as the monitoring
infrastructure.

Primary Keyword (Mandatory) Distributed data handling
Secondary Keyword (Optional) Object stores

Primary author

Alvaro Fernandez Casani (Instituto de Fisica Corpuscular (ES))

Co-authors

Andrea Favareto (Università degli Studi e INFN Genova) Carlos Garcia Montoro (Instituto de Fisica Corpuscular (ES)) Dario Barberis (Università e INFN Genova (IT)) Fedor Prokoshin (Federico Santa Maria Technical University (CL)) Javier Sanchez (Instituto de Fisica Corpuscular (ES)) Jose Salt (Instituto de Fisica Corpuscular (ES)) Julius Hrivnac (Universite de Paris-Sud 11 (FR)) Rainer Toebbicke (CERN) Ruijun Yuan (Laboratoire de l'Accelerateur Lineaire (FR)) Santiago Gonzalez De La Hoz (Instituto de Fisica Corpuscular (ES))

Presentation materials