Speaker
Description
The classification of HEP events, or separating signal events from the background, is one of the most important analysis tasks in High Energy Physics (HEP), and a foundational task in the search for new phenomena. Complex deep learning-based models have been fundamental for achieving accurate and outstanding performance in this classification task. However, the quantification of the uncertainty has traditionally been neglected when deep learning-based methods are used, despite its critical importance in scientific applications [1], [2].
In this work, we propose a Bayesian deep learning-based method for measuring uncertainty when classification of HEP events is performed using a deep neural network classifier. The work is focused on the use of the Monte Carlo Dropout (MC-Dropout) method, a variational inference technique proposed in [3] that is based on Dropout [4], the well-known regularization technique used to overcome overfitting. The Monte Carlo Dropout method allows production of the posterior distribution of the network weights by training a dropout network that approximates Bayesian inference. Thus, a Bayesian deep neural network considers a distribution over network parameters instead of a single point. The traditional dropout method randomly toggles off some neurons, with probability $D_{rate}$ during the training stage. However, the MC-Dropout method toggles off neurons both during the training stage and also during the inference stage.
In this work, we use the publicly available Higgs dataset described in [5]. This is simulated data, and the problem is to distinguish the signal from the background, where the signal corresponds to a Higgs boson decaying to a pair of bottom quarks according to the process: $gg \rightarrow H^0 \rightarrow W^{\mp} H^{\pm} \rightarrow W^{\mp} W^{\pm} h^0 \rightarrow W^{\mp} W^{\pm} b \bar{b}$. Furthermore, we plan to apply the proposed method using simulated data of the $\omega$ meson production off nuclear targets. Here, the problem is that the $\omega$ meson decays into four final-state particles: $\pi^+$ $\pi^-$ $\gamma$ $\gamma$, and the pions can also decay into muons and neutrinos, especially at low momentum [6].
The methodology of this work includes (i) training of Bayesian deep learning-based classifiers for the identification of signal and background (binary classification), using the Monte Carlo Dropout method, (ii) evaluate different $D_{rate}$; (iii) evaluate the classification performance; and (iv) compute three uncertainty measures including variance, mutual information, and predictive entropy. Preliminary results show on average 0.66 accuracy, 0.68 precision, 0.72 recall, and 0.70 F1 score, when a Monte Carlo Dropout model-based is used, with three hidden layers with 300 neurons each, and $D_{rate}=0.5$. We expect to increase the classification performance using hyper-parameters optimization, evaluating different network architectures, and varying the $D_{rate}$ parameter.
[1] Aishik Ghosh, Benjamin Nachman, and Daniel Whiteson. Uncertainty-aware machine learning for high energy physics. Phys. Rev. D, 104:056026, Sep 2021.
[2] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021.
[3] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
[4] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
[5] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1–9, 2014.
[6] Andrés Bórquez. The $\omega$ hadronization studies in the nuclear medium with the CLAS spectrometer. Master’s thesis, UTFSM, Valparaíso, Chile, 2021.
Significance
Machine learning-based systems perform an essential role in the analysis of huge volumes of data, in diverse scientific fields, including physics. More precisely, deep learning-based models have demonstrated outstanding results, however, accurate models are as important as the uncertainty quantification of the model prediction, and deep learning-based models only produce a point estimate which does not allow quantifying the uncertainty of each prediction.
Thus, in recent years, there has been increased interest within the scientific community to quantify the uncertainty related to machine learning models increasing the publications with the keywords machine learning, uncertainty quantification. In this work, we aim to measure uncertainty when event classification is performed using deep neural networks. We will apply the Monte Carlo Dropout method to measure epistemic uncertainty, including mutual information, variance, and predictive entropy measures, and hence to understand how the model makes decisions, an essential task in the scientific domain.
