The NOvA DAQ Monitor System

Not scheduled


1919-1 Tancha, Onna-son, Kunigami-gun Okinawa, Japan 904-0495
poster presentation Track1: Online computing


Susan Kasahara (University of Minnesota)


The NO$\nu$A (NuMI Off-Axis $\nu_{e}$ Appearance) experiment is a long-baseline neutrino experiment using the NuMI main injector neutrino beam at Fermilab and is designed to search for $\nu_{\mu}$ ($\bar{\nu}_{\mu}$) to $\nu_{e}$ ($\bar{\nu}_{e}$) oscillations. The experiment consists of two detectors; both positioned 14 mrad off the beam axis: a 220 ton Near Detector constructed in an underground cavern at Fermilab and a 14 kton Far Detector constructed in Ash River, MN, 810 km from the beam source. The detectors have similar design, and consist of planes of PVC extrusion cells containing liquid scintillator and wavelength shifting fibers. The fiber ends are readout by Avalanche Photodiodes (APDs). The primary task for the Data Acquisition (DAQ) system is to concentrate the data from the large number of APD channels (340000 channels at the Far Detector, 20000 channels at the Near Detector), buffer this data long enough to apply an online trigger, and record the selected data. The health and performance of the DAQ system is monitored with a DAQ Monitor system which monitors 180 custom data concentrator modules, over 200 buffer farm nodes, and 40 manager nodes at two detector sites separated by 810 km. The DAQ Monitor system is based on an open source third-party product, the Ganglia distributed monitoring system. Ganglia provides much of the functionality needed for the DAQ Monitor system “out-of-the-box” with the ability to collect standard computing performance metrics such as CPU usage, memory usage, and network transfer rates. Ganglia also provides the basis for displaying this information in a web display and storing this information in a database. We have augmented the Ganglia system for the specific needs of the NO$\nu$A DAQ Monitor system in the form of a custom metric client application interface which is used for purposes of constructing and distributing custom metrics by the components which make up the DAQ system. Using the custom metric client interface, monitored quantities specific to each DAQ component are sent at regular time intervals, displayed on the Ganglia web display and stored in the Ganglia database. Examples of custom metrics are trigger rates, data rates and sizes, and data corruption monitored rates. The Ganglia base has also been enhanced with a server side application used to read the monitored data from the Ganglia database, compare it to configurable thresholds, and issue warnings when monitored data falls out-of-range. The design of the NO$\nu$A DAQ Monitor system will be discussed as will experience with its deployment on the NO$\nu$A Near and Far detectors.

Primary author

Susan Kasahara (University of Minnesota)

Presentation materials

There are no materials yet.