Speaker
Description
The reliability and reproducibility of machine learning models are critically important for their use in automated systems. In the field of HEP, this may include detector optimization, use in blind analysis, and situations where estimates of model uncertainties are required. Building upon our previous research on developing robust model selection algorithms, we propose and comprehensively test an empirical approach to defining and automatically selecting robust deep learning models. In our study, a robust model is one that produces similar losses regardless of the choice of training sample from the population, and weight initialization. We previously implemented this approach in a regression task to reconstruct photon energy and position using GEANT4 simulation data from a Shashlik-type electromagnetic calorimeter. We investigated the impact of several factors on model robustness, including the size and heterogeneity of the training sample, the weight initialization method, and the inclusion of inductive biases (e.g., total cluster energy or its barycenter). In the present study, we demonstrate the universality of the proposed approach by extending its functionality to the classical, widely adopted computer vision task of classifying images from the CIFAR-10 dataset. The proposed method is a meta-algorithm with two key components: 1) a robustness assessment procedure that uses statistical analysis to evaluate loss variance across multiple, independently trained instances of the same model architecture and 2) a selection algorithm that sequentially filters less robust models out of a broad initial candidate pool by accumulating statistical evidence. Additionally, which is particularly important for HEP, the method allows one to extract the model's systematic uncertainties. Results confirm that the algorithm retains its efficiency, achieving convergence to a competitive model with a significantly lower total computational cost than an exhaustive search.