21-25 August 2017
University of Washington, Seattle
US/Pacific timezone

DAQExpert - An expert system to increase CMS data-taking efficiency

21 Aug 2017, 15:20
20m
Auditorium (Alder Hall)

Auditorium

Alder Hall

Oral Track 1: Computing Technology for Physics Research Track 1: Computing Technology for Physics Research

Speaker

Maciej Szymon Gladki (CERN, Geneva, Switzerland)

Description

The efficiency of the Data Acquisition (DAQ) in the new DAQ system of the Compact Muon Solenoid (CMS) experiment for LHC Run-2 is constantly being improved. A significant factor on the data taking efficiency is the experience of the DAQ operator. One of the main responsibilities of DAQ operator is to carry out the proper recovery procedure in case of failure in data-taking. At the start of Run-2, understanding the problem and finding the right remedy could take a considerable amount of time, sometimes up to minutes. This was caused by the need to manually diagnose the error condition and to find the right recovery procedure out of an extended list which changed frequently over time. Operators heavily relied on the support of on-call experts, also outside working hours. Wrong decisions due to time pressure sometimes lead to an additional overhead in recovery time.

To increase the efficiency of CMS data-taking we developed a new expert system, the DAQExpert which provides shifters with optimal recovery suggestions instantly when the failure occurs. This tool significantly improves the response time of operators and the success rate of recovery procedures. Our goal is to cover all known failure conditions and to eventually trigger the recovery without human intervention wherever possible. This paper covers how we achieved two goals - making CMS more efficient and building a generic solution that can be used in other projects as well. More specifically we discuss how we: determine the optimal recovery suggestion, inject expert knowledge with minimum overhead, facilitate post-mortem analysis and reduce the amount of calls to on-call experts without deterioration of CMS efficiency. DAQExpert is a web application analyzing frequently updating monitoring data from all DAQ components and identifying problems based on expert knowledge expressed in small, independent logic-modules written in Java. Its results are presented in real-time in the control room via a web-based GUI and a sound-system in a form of short description of the current failure, and steps to recover. Additional features include SMS and e-mail notifications and statistical analysis based on reasoning output persisted in a relational database.

Primary authors

Jean-Marc Olivier Andre (Fermi National Accelerator Lab. (US)) Ulf Behrens (Deutsches Elektronen-Synchrotron (DE)) James Gordon Branson (Univ. of California San Diego (US)) Olivier Chaze (CERN) Sergio Cittolin (Univ. of California San Diego (US)) Cristian Contescu (Fermi National Accelerator Lab. (US)) Georgiana Lavinia Darlea (Massachusetts Inst. of Technology (US)) Christian Deldicque (CERN) Zeynep Demiragli (Massachusetts Institute of Technology, Cambridge, Massachusetts, USA) Marc Dobson (CERN) Nicolas Doualot (Fermi National Accelerator Lab. (US)) Samim Erhan (University of California Los Angeles (US)) Jonathan Fulcher (CERN) Dominique Gigi (CERN) Maciej Szymon Gladki (CERN, Geneva, Switzerland) Frank Glege (CERN) Guillelmo Gomez Ceballos Retuerto (Massachusetts Inst. of Technology (US)) Jeroen Hegeman (CERN) Andre Holzner (Univ. of California San Diego (US)) Mindaugas Janulis (Vilnius University (LT)) Lorenzo Masetti (CERN) Frans Meijers (CERN) Emilio Meschi (CERN) Remi Mommsen (Fermi National Accelerator Lab. (US)) Srecko Morovic (CERN, Geneva, Switzerland) Vivian O'Dell Luciano Orsini (CERN) Christoph Paus (Massachusetts Inst. of Technology (US)) Petia Petrova (CERN) Andrea Petrucci (Rice University (US)) Marco Pieri (Univ. of California San Diego (US)) Dinyar Rabady (CERN) Attila Racz (CERN) Thomas Reis (CERN) Hannes Sakulin (CERN) Christoph Schwick (CERN) Dainius Simelevicius (CERN, Geneva, Switzerland) Michail Vougioulas (CERN, Geneva, Switzerland) Petr Zejdl (Fermi National Accelerator Lab. (US)) Philipp Maximilian Brummer (CERN)

Presentation materials

Peer reviewing

Paper