Speaker
Susan Kasahara
(University of Minnesota)
Description
The NO$\nu$A (NuMI Off-Axis $\nu_{e}$ Appearance) experiment is a long-baseline neutrino experiment using the NuMI
main injector neutrino beam at Fermilab and is designed to search for $\nu_{\mu}$ ($\bar{\nu}_{\mu}$) to $\nu_{e}$
($\bar{\nu}_{e}$) oscillations. The experiment consists of two detectors; both positioned 14 mrad off the beam axis: a
220 ton Near Detector constructed in an underground cavern at Fermilab and a 14 kton Far Detector constructed in Ash
River, MN, 810 km from the beam source. The detectors have similar design, and consist of planes of PVC extrusion
cells containing liquid scintillator and wavelength shifting fibers. The fiber ends are readout by Avalanche
Photodiodes (APDs). The primary task for the Data Acquisition (DAQ) system is to concentrate the data from the large
number of APD channels (340000 channels at the Far Detector, 20000 channels at the Near Detector), buffer this data
long enough to apply an online trigger, and record the selected data.
The health and performance of the DAQ system is monitored with a DAQ Monitor system which monitors 180 custom data concentrator modules, over 200 buffer farm nodes, and 40 manager nodes at two detector sites separated by 810 km. The DAQ Monitor system is based on an open source
third-party product, the Ganglia distributed monitoring system. Ganglia provides much of the functionality needed for
the DAQ Monitor system “out-of-the-box” with the ability to collect standard computing performance metrics such as CPU
usage, memory usage, and network transfer rates. Ganglia also provides the basis for displaying this information in a web
display and storing this information in a database.
We have augmented the Ganglia system for the specific needs of
the NO$\nu$A DAQ Monitor system in the form of a custom metric client application interface which is used for purposes
of constructing and distributing custom metrics by the components which make up the DAQ system. Using the custom
metric client interface, monitored quantities specific to each DAQ component are sent at regular time intervals,
displayed on the Ganglia web display and stored in the Ganglia database. Examples of custom metrics are trigger
rates, data rates and sizes, and data corruption monitored rates. The Ganglia base has also been enhanced with a
server side
application used to read the monitored data from the Ganglia database, compare it to configurable thresholds, and
issue warnings when monitored data falls out-of-range.
The design of the NO$\nu$A DAQ Monitor system will be discussed as will experience with its deployment on the NO$\nu$A Near and
Far detectors.
Primary author
Susan Kasahara
(University of Minnesota)
