Choose timezone
Your profile timezone:
Alice Weekly Meeting: Software for Hardware Accelerators
→
Europe/Zurich
ALICE GPU Meeting
-
-
Speaker: Christian Sonnabend (CERN, Heidelberg University (DE))
Framework
- ONNX: Non-deterministic compute on GPU
- More global issue. Tried all variations, but even in float32 using
(mPImplOrt->sessionOptions).AddConfigEntry("session_options.use_deterministic_compute", "1");
the compute was not deterministic. - Investigation in GitHub threads and forums shows that this is indeed not guaranteed by ONNX (or any other framework), not even during training!
- SetDeterministicCompute: "... If set to true, this will enable deterministic compute for GPU kernels where possible. ..."
- More global issue. Tried all variations, but even in float32 using
- Variations (using only the classification network with fixed threshold):
- L2, N16: 276782342 \pm 105 ~ 3.8x10e-5% deviation
- L5, N128: 265601210 \pm 111 ~ 4.2x10e-5% deviation
GPU timing test
(see debugLevel=0 folder) Classification network + CF regression-> Larger batch size is better, but also: Almost no difference in wall time from L2_N16 to L5_N32 for almost any batchsize -> L5_N32 will be the network of choice(see debugLevel=1 folder) Classification network + CF regression-> Larger batch size is better. Increase from L2_N16 to L5_N32 is factor 1.4 but typically largest jumps noticeable from 32 to 64 neurons per layer (potentially connected to cross-warp computations?)Memory consumption increases with larger batchsize: Input = BS x sizeof(float16) x (e.g. 3x9x9 + 3) x #streams + ONNX. For batchsize of 262144, 3 streams: Input ~387MB + ONNXPhysics- Did the QA on the ideal clusters again now compared to ~1.5 years ago
- Changes in between:
- Added looper tagging
- Changed peak finder to the peak finder of the GPU cluster finder (for training data generation)
- Using native clusters from reconstruction, not the clusters generate in the QA macro
- Using simulation which enforces 0-5% centrality, so efficiencies go down a bit, fake-rates go up a bit
- Network size = classification network size, Regression algorithm = GPU CF regression
Raw efficiency without loopers: How many clusters have been attached to at least one ideal (MC) cluster, removing all (local) regions in loopers
- NN can never attach more clusters than GPU CF because it won't produce more clusters (it can only reject)
Another possibility: #(attached ideal clusters, not in looper tagged region) / #reconstructed clusters
- Can be artificially inflated with number of loopers, but in all cases network should reject looping clusters better, so efficiency should go up
To-Do
- Create standalone reproducer for ONNX non-determinism
- Create Efficiency-Fake plots with regression network
- ONNX: Non-deterministic compute on GPU
-
Speaker: Dr Oliver Gregor Rietmann (CERN)
Summer Student Milla Bramsted
- She is working on benchmarking SoA code on GPUs
- We will track her project in this google doc
- She is adding CUDA kernels to this repo
- Three CUDA kernels are now running in our benchmark framework
- Next steps:
- Run the same kernels for different data layouts
- Add the other NGT SoA approaches again (were temporarily removed)
ALICE O2 CI-Pipelines on NGT Cluster
- A fork of the AliceO2 repo is not in the NextGenTrigggers (NGT) GitHub organization
- It has a GitHub action running the standalone benchmark on NGT GPUs (H100)
- Uses the builds in /cvmfs/alice.cern.ch/ of O2 and dependencies (pipeline takes about 7 minutes)
- Different GPUs are tested in parallel on different VMs
- O2 standalone benchmarks now works on the following ngt-resources:
- Nvidia H100 188GB NVL
- AMD Instinct MI300X
- AMD Radeon Pro W7900: Works, but is a bit hacky, need to improve
- Nvidia L40S: Works fine, but needs to be added to the CI
- Next steps:
- Find a better solution for W7900
- Add L40S to CI (trivial)
- Avoid some code repetition in the github workflow file
Implement NGT SoA Code in O2 standalone benchmark
- Working on this fork of the AliceO2 repo
- Simplified and optimized the SoA code in the last few weeks
- Everything is running and we have identified the classes apply our SoA code to
- Next steps:
- Discuss these classes with David
- Implement the SoA code on those classes
-
