We had a long
discussion about how to
proceed with sampler comparisons, and how to present results in the paper. A summary of decisions is as follows:
1) We decided to write a first paper on optimisation techniques, and leave posterior samplers to a quick follow up. The author list for both will be the same.
Those who have coded up posterior sampling techniques should pick up an optimisation technique for the first paper.
The pygmo package (see the slides) has a list of algorithms implemented, and you should please volunteer to take one of those and perform the optimisation as described below. The algorithm should not overlap with one already in the list of implemented algorithms (also available in the slides).
This will allow us to get the optimisation paper out before the DarkMachines workshop at CERN.
3) Luc will code up 4 test functions that represent different physics cases (e.g. a unimodal distribution, a multimodal distribution, a broad minimum with a strongly peak, off-centered global minimum and
a high-dimensional function). These will be compiled and distributed as binary files, so that we don't know what they are (this removes the ability to set hyperparameters of sampling algorithms based on knowledge of the function, which is relevant for some samplers).
These should be available by the middle of next week, and we all need to test them and respond to Luc on a short timescale in case he needs to compile binaries on other systems for compatibility. The functions will be integrated with Bob's framework by Bob.
For each test function, the “owner” of each algorithm should try and find the optimum settings that give the best likelihood in the smallest number of likelihood evaluations.
It is up to you how to do this (intuition plus trial and error is probably the main method).
An appendix in the paper will contain tables of the best settings that were found for each algorithm for each function.
A table in the main text will then compare the algorithm performance for the optimum settings.
Columns should include the best fit likelihood found, the number of likelihood evaluations required to get it,
whether the algorithm is parallelisable.
By running all of the algorithms on a fixed number of cores on the same machine, we will also add a column that gives the wall time for each algorithm (since this is sensitive to any extra overhead that
each algorithm produces between likelihood evaluations).
5) When comparing algorithms in the paper, we should also
(if indeed there is a default), and h
ow much variability in the results
occurred when optimising.
The aim is for the paper to represent the typical performance expected in the use case “physicists grabs the algorithm to solve a problem, and has limited time and resources to optimise it”.
Next steps: Bob to merge remaining pull requests,
Luc to provide functions, users to test functions and optimise their algorithms.
We should aim to have results to discuss in 2-3 weeks time, then the next meeting will involve discussion of the results (please add these to the paper draft). If we find that this approach is deficient in practise, we will use the meeting to fix it.