Speakers
Description
The increasing reliance on machine learning (ML) and particularly deep learning (DL) in scientific and industrial applications requires models that are not only accurate, but also reliable under varying conditions. This is especially important for automated machine learning and fault-tolerant systems where there is limited or no human control. In this paper, we present a novel, task-independent approach for assessing the robustness of machine learning models. Our methodology quantifies model robustness across different training samples and weight initialisations, using statistical measures of test loss variability. In addition, we propose a meta-algorithm for selecting reliable models from a set of candidates, balancing performance and robustness. We apply our approach to deep learning architectures with a small number of convolutional and fully connected layers, allowing us to efficiently explore thousands of configurations. Applying this method we have identified robust models for two regression problems by investigating the effects of training sample size, weight initialisation, and inductive bias. Our results show that model robustness depends on whether raw or high-level features are used, and we show that incorporating inductive bias can reduce training time and prediction performance without degrading model robustness. The proposed model robustness measurement and selection strategies can be integrated into existing AutoML systems, providing a novel approach to automated and robust model development for high-risk environments such as scientific instrument design and complex data-driven workflows.
Significance
This work introduces a general-purpose approach for evaluating and selecting robust machine learning models, with direct relevance to automated and fault-tolerant systems. It fills a gap in current research by addressing robustness across data sampling and initialisation.
References
"Stable neural network models for calorimeter optimization" on Fourth MODE Workshop on Differentiable Programming for Experiment Design https://indico.cern.ch/event/1380163/contributions/6061793/
| Experiment context, if any | This work uses data based on stand-alone simulations that generically reproduce the LHCb electromagnetic calorimeter; however, the methodology is completely generic and does not depend on the specificity of this dataset. The approach can be applied to any suitable data, regardless of its origin. |
|---|