Alice Weekly Meeting: Software for Hardware Accelerators

→ Europe/Zurich
Zoom Meeting ID
61230224927
Host
David Rohr
Useful links
Join via phone
Zoom URL
    • 10:00 AM → 10:20 AM
      Discussion 20m
      Speaker: David Rohr (CERN)

      Color code: (critical, news from this week: blue, news from last week: purple, no news: black)

      Sync reconstruction

       

      Async reconstruction

      • Need to investigate short GPU stall problem.
      • Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
      • Will tune existing 16-core settings, add a SITEARCH for 16core CPU, and 16coreCPU + generic NVIDIA / AMD GPU, like for 8 core.
      • Will retune EPN async workflow for TPC + ITS on GPU on 2025 data.

       

      GPU ROCm / compiler topics:

      • Problem with building ONNXRuntime with MigraphX support.
      • Need to find a way to build ONNXRuntime with support for CUDA and for ROCm.
      • Try to find a better solution for the problem with __device__ inline functions leaking symbols in the host code.
      • First version of ALIBUILD_O2_FORCE_GPU=build env variable option to use CUDA / ROCm from alidist merged.
        • Working nicely so far.
        • Out of the 2 bugs reported to ONNXRuntime for CUDA architectures, 1 is no fixed upstread, we can remove the workaround after the next update.
        • ALIBUILD_O2_FORCE_GPU=ci now defaults to "build" for SLC10. slc10-gpu-ci is now working in this mode, and seems good. (only thing still to be done: need to check that the architecture override is now fixed correctly after Giulio's last change).
        • Next to do: Move FullCI to SLC10 for O2 and alidist: Then we have all CI GPU builds on SLC10.
      • CUDA 13.4 released, supports GCC 16, will bump in cuda.sh recipe, but for now let's first stabilize the CI with the build mode.
      • Did a quick validation of ROCm10, seems as stable as 7.14, but still need to do long-term test.

       

      TPC / GPU Processing 

      • WIP: Use alignas() or find a better solution to fix alignment of monte carlo labels: https://its.cern.ch/jira/browse/O2-5314
      • Need to check the problem with ONNX external memory allocator.
      • Next high priority topic: Improvements for cluster sharing and cluster attachment at lower TPC pad rows. PR: https://github.com/AliceO2Group/AliceO2/pull/14542
      • Check for unnecessary f64 instructions in GPU code.

       

      Other topics:

      • GCC bump - Status?
      • We can go to GCC 16, once we switch the FullCI to SLC10 and the new GPU runtime build mode.

       

      EPN GPU Topics:

      • Discuss with Ernst / Giulio how to do a build for SLC10 on EPNs with build mode.
    • 10:20 AM → 10:25 AM
      TPC ML Clustering 5m
      Speaker: Christian Sonnabend (CERN, Heidelberg University (DE))
    • 10:25 AM → 10:30 AM
      GPU Parameter Optimizations 5m
      Speaker: Gabriele Cimador (CERN, Università and INFN Torino)

      EB approved proceedings yesterday (just minor corrections to apply)

      What should I do:

      1. Send an email to conference organizers with corrected version
      2. Wait for first round of review to apply also these minor modifications
    • 10:30 AM → 10:35 AM
      Efficient Data Structures 5m
      Speaker: Dr Oliver Gregor Rietmann (CERN)
       

      NextGenTrigger Task 1.7

      • Nothing

       

      Implement NGT SoA Code in O2 standalone benchmark

      • Working on this fork of the AliceO2 repo, with a CI pipeline:
        • Running on NGT hardware with 4 different GPUs (Nvidia and AMD)
        • Extended CI-pipline to fail if GPU.out changes
      • Implemented SoA in SectorTracker: GPUTPCBaseTrackParam, GPUTPCTrackParam, GPUTPCTracklet, GPUTPCTrack

       

      • Finished CHEP2026 proceedings
      • Make better use of SoA to improve performance
        • TrackletConstructor
          • Register spill from rowHits array, but the accesses are not coalesced. (UpdateTracklet)
          • global memory loads on average use only 4.4 out of 32 bytes sector
          • active warps per thread 9.34 out of 32 (29%) due to different tracklet lengths
        • ExtrapolationTracking
          • Register spill from rowHits array, but the accesses are not coalesced. (UpdateTracklet)
          • (from UpdateTracklet) 
          • global memory loads on average use only 4.4 out of 32 bytes sector
          • active warps per thread 2.65 out of 32 (8%) due to nested if conditions (and tracklet length)
      • Next Steps
        • Fix coalescing by making the threads of a warp process the same rows
        • Fix thread occupancy by ordering tracks per length
        • Fix warp divergence in ExtrapolationTracking by determining active threads before kernel launch.
        • (Generate GPU parameters .csv file for SoA and to see if the optimal parameters are different)
    • 10:40 AM → 10:45 AM
      TPC Clusterization / OpenCL / Highly Ionizing Particles 5m
      Speaker: Felix Weiglhofer (CERN)
       
       

      OpenCL

      No news.

      GPU Servers

      • CI Server: Working on test for deterministic mode
      • pdp partition: Jobs now default to a time limit of 8 hours to prevent accidentally blocking the nodes. (For long running jobs: Use --time, no upper time limit)

       

      Highly Ionizing Particles

      • Working on fixing the time position: 
        • Position now in middle of the saturated plateau
        • Only marginally improves tracking (still losing ~6% of tracks with filter)
    • 10:45 AM → 10:50 AM
      ITS Tracking 5m
      Speakers: Felix Schlepper (CERN, Heidelberg University (DE)), Gabriele Cimador (CERN, Università and INFN Torino), Matteo Concas (CERN)

      Gabriele: 
      New seeding vertexer PR at https://github.com/AliceO2Group/AliceO2/pull/15733
      Next steps:

      • Profile GPU kernels
        • Maybe do a manual tuning
      • Further optimize kernels (e.g. shared memory / warp reductions ... ) 
      • Benchmark and optimize CPU version (real 23kHz @ 5 threads)
      • Investigate SoA vs AoS (Oliver's library)

       

      Felix: nothing

    • 10:50 AM → 10:55 AM
      System Run Coordination Topics 5m
      Speaker: Ernst Hellbar (CERN)