Color code: (critical, news from this week: blue, news from last week: purple, no news: black)

Sync reconstruction

Async reconstruction

• Need to investigate short GPU stall problem.
• Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
• New GPU architecture selection for async in O2DPG looks good, should be merged.
• Test with GPU GRID jobs at NERSC pending.
• Asked DPG to run first test with ITS tracking on GPU on EPNs.

GPU ROCm / compiler topics:

• Issues that disappeared but not yet understood: random server reboot with alma 9.4, miscompilation with ROCm 6.2, GPU getting stuck when DMA engine turned off, MI100 stalling with ROCm 5.5.
• Problem with building ONNXRuntime with MigraphX support.
• Need to find a way to build ONNXRuntime with support for CUDA and for ROCm.
• Try to find a better solution for the problem with __device__ inline functions leaking symbols in the host code.
  
• LLVM Bump to 20.1: status?
• Miscompilation / internal compiler error fixed in new clang for ROCm 7.x, SDMA engine synchronization bug still not fixed.
  • Compilation errors fixed in ROCm trunk, to go into ROCm 7.
  • Validated standalone benchmark in deterministic mode, and ran FST.
  • Performance regression of ~1.5% without RTC, no regression with RTC. Reported to AMD. Not clear if this will be fixed.
  • Serialization bug on MI100 still not fixed, still requires the workaround.

TPC / GPU Processing 

• WIP: Use alignas() or find a better solution to fix alignment of monte carlo labels: https://its.cern.ch/jira/browse/O2-5314
• Waiting for TPC to fix bogus TPC transformations for good, then we can revert the workaround.
  
• Waiting for TPC to check PR which uses full cluster errors including average charge and occupancy map errors during seeding.
  
• Final solution: merging transformation maps on the fly into a single flat object:
  • Compilation issues on GPU fixed.
  • Matthias did some checks, found two minor bugs to be fixed by Sergey.
  • Speed seems ok, 0.25s for merging 2 maps.
• Pending OpenCL2 issues:
  • printf not working due to confirmed bug in clang, fix is being prepared. Prevents further debugging for now.
    
  • Crash in merger, which can be worked around by disabling clang SPIRV optimization. Probably bug in clang, but need to fix printf first to debug.
    
  • Also with optimization disabled, crashing later in TPC merging, need printf to debug.
• Need to check the problem with ONNX external memory allocator.
• Next high priority topic: Improvements for cluster sharing and cluster attachment at lower TPC pad rows. WIP PR: https://github.com/AliceO2Group/AliceO2/pull/14542
  • Now has general improvements in all aspects: higher efficiency, lower clone rate, more clusters rejected online, better cluster to track association, faster on AMD GPUs (though slightly slower on NVIDIA and CPU).
  • Trying to understand one problem found by Ruben where it gets stuck processing a TF, then we might merge it to benefit from the improevements.
  • Further developments in different branch on top.
• Bug in multi-threaded pipeline when timeframes do not arrive in order. Happened again, Ernst and Giulio obtained a stack trace and attached a gdb. Not fully clear why, but the completion policy is stuck. Tentative fix for a possible deadlook between the completion policy and DPL is here https://github.com/AliceO2Group/AliceO2/pull/14640. Need to test online if the problem goes away.
• gpu-reconstruction quitting with error in some async jobs due to running out of buffers, need to check but probably bad TPC data.

EPN GPU Topics: