Color code: (critical, news from this week: blue, news from last week: purple, no news: black)
Sync reconstruction
Async reconstruction
- Need to investigate short GPU stall problem.
- Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
- Will tune existing 16-core settings, add a SITEARCH for 16core CPU, and 16coreCPU + generic NVIDIA / AMD GPU, like for 8 core.
- Will retune EPN async workflow for TPC + ITS on GPU on 2025 data.
GPU ROCm / compiler topics:
- Problem with building ONNXRuntime with MigraphX support.
- Need to find a way to build ONNXRuntime with support for CUDA and for ROCm.
- Try to find a better solution for the problem with __device__ inline functions leaking symbols in the host code.
- First version of ALIBUILD_O2_FORCE_GPU=build env variable option to use CUDA / ROCm from alidist merged.
- Working nicely so far.
- Out of the 2 bugs reported to ONNXRuntime for CUDA architectures, 1 is no fixed upstread, we can remove the workaround after the next update.
- ALIBUILD_O2_FORCE_GPU=ci now defaults to "build" for SLC10. slc10-gpu-ci is now working in this mode, and seems good. (only thing still to be done: need to check that the architecture override is now fixed correctly after Giulio's last change).
- Next to do: Move FullCI to SLC10 for O2 and alidist: Then we have all CI GPU builds on SLC10.
- CUDA 13.4 released, supports GCC 16, will bump in cuda.sh recipe, but for now let's first stabilize the CI with the build mode.
- Did a quick validation of ROCm10, seems as stable as 7.14, but still need to do long-term test.
TPC / GPU Processing
- WIP: Use alignas() or find a better solution to fix alignment of monte carlo labels: https://its.cern.ch/jira/browse/O2-5314
- Need to check the problem with ONNX external memory allocator.
- Next high priority topic: Improvements for cluster sharing and cluster attachment at lower TPC pad rows. PR: https://github.com/AliceO2Group/AliceO2/pull/14542
- Check for unnecessary f64 instructions in GPU code.
Other topics:
- GCC bump - Status?
- We can go to GCC 16, once we switch the FullCI to SLC10 and the new GPU runtime build mode.
EPN GPU Topics:
- Discuss with Ernst / Giulio how to do a build for SLC10 on EPNs with build mode.