Under-coverage in high-statistics counting experiments with finite MC samples

by Cristina Alexe (Scuola Normale Superiore & INFN Pisa (IT))

Wednesday 8 Oct 2025, 16:00 → 17:30 Europe/Zurich

Abstract:

We consider the problem of setting confidence intervals on a parameter of interest from the maximum-likelihood fit of a physics model to a binned data set with a large number of bins, large event-counts per bin, and in the presence of systematic uncertainties modeled as nuisance parameters. We use the profile-likelihood ratio for statistical inference and focus on the case in which the model is determined from Monte Carlo simulated samples of finite size. We start by presenting a toy model in which the asymptotic properties of the profile-likelihood ratio are manifestly broken even if the numbers of events per bin in both the data and simulated samples are seemingly large enough to warrant their validity. We then move to the general setting to show how statistical uncertainties in the Monte Carlo predictions can affect the coverage of confidence intervals constructed in the asymptotic approximation always in the same direction, namely they lead to systematic under-coverage.

Speaker Bio: 

Cristina Alexe is a PhD student working on the measurement of the W boson mass at the CMS Experiment at CERN. She joined Scuola Normale Superiore di Pisa in 2022 after completing a Master’s degree in Theoretical Physics at The University of Manchester, with a thesis on the statistical combination of CP-violation measurements in the context of a new theoretical framework. Cristina is particularly interested in how we use statistics to take decisions, from physicists claiming results to governments making policies.

https://arxiv.org/abs/2401.10542