July 31, 2018 to August 6, 2018
Maynooth University
Europe/Dublin timezone

Pseudosignificances as figures of merit: a systematic study and Bayesian solutions

Aug 1, 2018, 2:40 PM
Hall C (Arts Bldg.)

Hall C

Arts Bldg.

Talk H. Statistical Methods for Physics Analysis in the XXI Century Statistical Methods for Physics Analysis in the XXI Century


Dr Pietro Vischia (Universidad de Oviedo (ES))


Optimization problems in HEP often involve maximizing a measure of how sensitive is a given analysis to an hypothesis with respect to another hypothesis; the latter is referred to as "null" hypothesis and in a frequentist framework is tested against the former, which is referred to as "alternative" hypothesis.

In most cases, it is desirable to fully compute the expected frequentist significance, accounting for all sources of systematic uncertainty and interpreting the result as the real sensitivity of the analysis to the effect sought.
Sometimes, however, either computational or conceptual reasons can favour the use of different or approximate figures of merit, often collectively called "pseudosignificances", which exhibit different properties depending on the relationship between the hypotheses being tested.

This work will review the most common definitions of sensitivity (pseudosignificances), and compare them with the fully frequentist significances computed in toy analyses spanning a spectrum of typical HEP use cases. A connection will be made with the concept of Bayes Factor, and evidence values from Bayesian significance tests will be studied and evaluated in the same toy cases, attempting to build an improved approximate condition-aspecific figure of merit. Finally, Bayesian solutions to the on-off problem, well known in astrophysics, will be transported to the typical HEP cases.

Primary author

Dr Pietro Vischia (Universidad de Oviedo (ES))

Presentation materials