Choose timezone
Your profile timezone:
HEPiX Benchmarking Working Group
Attendees
- in the CERN room: Olga V., Andrea V., Luca A., Domenico G.
- from Vidyo: Michele M, Martina J., Manfred A., Martin B.
Today we have 3 presentations on the work done in August. Another presentation from Chris Hollowell has been postponed to next meeting:
Presentation from Andrea Valassi on general work for the benchmarking suite to exploit the commonalities among all the benchmark, in particular for the HEP Application ones. This should improve robustness and error handling of the Continuos Integration and individual bmk
Now there is common bmk driver so it is easier to debug silent failure and handle exit code. One benefit is that the validation of the json summary and a consistent way to use the command line flags
All this work started from the need to define a common Json schema but first some work was needed. Now the common schema is been defined (Martina is working on this). Then from the several json we can produce the HEPSCORE (or several type of HEPSCORE per type of workload, if needed)
There are still some bugs that need to be fixed e.g. the atlas digi reco score is not the sum of the sub-benchmark score but should be computed on the sum of times or in the total time.
There is also a problem with usage of different version of python.
The CI contains several level of virtualization. This new version has less level and those level are independent to only the level that need rebuild are actually rebuilt and this has benefit also in people running different benchmark on the same server. Andrea will provide a picture of the layers for reference to be used at pre-GDB and future paper or presentation
Also now benchmarks runs and non-root but as "bmkuser"
Other work is going on the licensing for distribution to commercial companies, GPU workloads, vectorization, HPC
Michele slides on Robustness
Running on a small server that has some peculiarity:
1) a number of core not multiple of 4 (6 core)
2) only 1,33 GB per LogicalCPU (16 GB / 12 Lcpu)
ALICE GEN sim now ok
ATLAS GEN SIM crash when run 12 copies / 1 thread but ok with 6c/2t or 4c/3t. Running MP permits to decrease the mem footprint and while with 6c/2t we are immediately under the saturation with 12c/1t we exceed. We should try with a machine with more memory
ATLAS DIGI RECO failing with memory problem OOM_KILLED as already reported by Andra Valassi. Again a problem of memory but first we should fix two problem
- The score must be computed on the tput/total time, not a sum of the 4 scores
- We should move it to MP framework to decrease the amount of memory to see it this improves the memory footprint
Domenico suggests to split the DIGI and RECO steps (that are independent) in two separate containers, to check which of the two causes the problem.
Manfred slides on reproducibility
In the past we observed order of 20% spread in result with ALICE gen-sim, ATLAS Sim and Atlas digi-reco
Now if Manfred runs consecutively 20 times a workload, and takes them 3 by 3 and takes the median value this spread in reduced from 20% to 1% on ALICE SIM while ATLAS Gen remains high and DIgI-Reco still in progress. A discussion follow on the property of this kind of statistics. Manfred will provide the json to see the distributions of results and compare them with other workload
It was noticed that the approach of using sliding groups of 3 causes a bias because the same numbers are used at least 2 (3) times in each subgroup. It is suggested to split the 20 runs in not overlapping groups of 3 to evaluate the spread
Licensing
We would like to distribute the HEP application in binary form + some python scripts + other script to people that want to make benchmark. The previous HEP Benchmark consisted in a small script that was running software licensed from SPEC so each WLCG site, HEP or other community site has to procure a SPEC license and each vendor who wanted to participate to a tender had to do the same.
We will contact the experiments to clarify their licensing
Michele will take care of the BMK-175 Jira ticket.
Domenico will contact the CERN Office that is concerned with this.
There are minutes attached to this event.
Show them.