Sergei, News

• Next meeting (January): multi-class and multi-objective classification/regression, catch-all, workshop reports 
  • Contrary to what we announced in November, the topic is not going to be a tagging.  We plan to have a dedicated workshop (see below), due to the large number of proposed contributions
• Community White Paper (CWP) of Hep Software Foundation to be produced by sometime next summer, includes machine learning. IML should contribute to that.
• IML Workshop planned for March 20-22 (see slides for details)
  • Tagging workshop (with hands-on) session
  • CWP discussion
  • Tutorials

Enrico Guiraud, Generative models and EM algorithm 

• Introduction to generative models
• Main ingredients
  • Latent variables (not observed, e.g. the model is made by 3 gaussians the parameters of the individual gaussians are the latent variables)
  • Observed variables
  • Model parameters
• Methematically: conditional probability of observed data depending on hidden states (latent variables) and parameters
  • Can be treated as a likelihood maximization problem
  • Directly maximizing the log likelihood very difficult/impossible due to the large sum over hidden states
  • EM algorithm: workaround the problem maximizing the "free energy", obtained introducing variational distributions q(s)
  • Details on the choice of the q(s) are discussed in the slides
• An example is shown with the "noisy-OR" model
• A more complex example is applied to the Mnist data set
  • Recovers "digits"-like latents
• Difference with respect to other approaches: the model is explicit (you have to write a model in terms of hidden variables)
• Questions
  • Kyle: Latent variables in HEP are Monte Carlo truth, but much more complex: millions of hidden variables in geant simulation
    • Possibility to merge to kinds of generative models, non-explicit for simplified model of the detector, and EM minimization to get access to hidden variables
    • Does not replace a full simulation, but it could provide insight on a few carefully chosen hidden variables
  • Sergei: can think about using this for a fast simulation.

Gilles Louppe, Learning to Pivot with adversaria networks

• How to use a generative model to constrain a classifier
• One of the typical problems in phsyics: how to incorporate/treat systematic uncertainties coming from the model uncertainties
• Goal: find a classifier which is not sensitive to systematic variations of nuisance parameters
• Slide 4: it means finding a classifier f which is a "pivotal quantity"
• 2 networks:
  • one is the classifier
  • one is the adversary, which produces the posterior value of the nuisance parameters based on the output of the classifier
    • If the adversary can produce a meaningful posterior of z it means that the classifier depends on the nuisance parameter
    • Want to make the adversary very bad
• Details on the architecture shown in the slides
• Strategy: minimize a loss function built with the loss function of the classifier minus the loss function of the adversary
  • (Proof on the mini-max optimization and minimization algorithm shown in the slides)
  • Weight of the adversary can be controlled by a parameter λ, controls the trade off between accuracy and robustness
• Toy example discussed in the slides: two classes (gaussians) where the exact relative position is not known 
  • Shows that the method works and that the robustness comes at the price of poorer classification performance
• HEP-inspired example also shown (W and QCD jets discrimination)
  • Nuisance: pileup (extreme cases of 0 or 50 pileup events considered)
• Slide 17: optimize λ with respect to some other objective, e.g. the final median statistical significance of the signal
• Questions
  • Sergei Slide 18: what are the bands? Experiment repeated multiple times.
    • interesting to observe that the treshold changes
    • Yes, but this is effectively a different classifier.
  • Sergei: What do the curves look like for  10< λ < 500, same general shape?
    • Answer: looked at it, didn’t show but generally yes
  • Sergei: did you try with multiple nuisance parameters? Not yet, but extension should be trivial
  • Tatiana: What's the proportion of events with 0 and 50 pile-up events? 50-50
    • It seems the result does not depend on whether the training is done with the Z=1 or Z=0 class
    • Question deals with robustness of the result to various levels of pile-up
  • Tatiana: Did you compare with the approach of Mike Williams (uBoost https://arxiv.org/abs/1305.7248)? no
    • one difference: this method was developed for non-observable parameters.
    • Subsequent discussion focused on whether the choice of pile-up as a nuisance parameter was justified.

Mike Williams, Event generator tuning using Bayesian Optimization

• Introduction on what Bayesian optimization is
• Investigate if it can be used for MC tuning -> generate posterior distributions for Pythia parameter based on data
• This method can automatically assign uncertainties to the parameters
• Use "Monash" as true data, see if they can recover Monash parameters via bayesian optimization
• Slide 3: if a parameter is not really constrained by a particular observation it correctly gets a huge error bar
  • Convergence is relatively fast (order of 50 queries * nParameters)
• Global fit of all 20 parameters is less precise than "block" training, but still very satisfactory
• Potential improvements discussed on the slides: expert knowledge, knowledge transfer, pre-simulation of small samples, treatment of discrete parameters, extension to larger parameter spaces
• Questions
  • Sergei : bayesian optimization of ML hyperparameters, is there anybody who tried that? 
    • Baldi et. al.  
    • Some other people did it, for instance there is an entry on Tim Head's blog
  • Sergei: Does the tool (spearmint) work in parallel? Yes, but few choiches on how  parallelize (e.g. pythia -> trivial parallelization). The tool allows for parallel optimization with different data set, useful if you have distributed resources.
  • Sergei: Next step? Will you try this with real data?
    • Yes, will be tested on real data. Was tested on ee, finds better chi2 than default tune, but some parameters seem unphysical

Jonah Bernhard,  Applying Bayesian parameter estimation to relativistic heavy-ion collisions

• Postponed due to technical problems with Vidyo