Alice Weekly Meeting: Software for Hardware Accelerators

→ Europe/Zurich
Zoom Meeting ID
61230224927
Host
David Rohr
Useful links
Join via phone
Zoom URL
    • 10:00 → 10:20
      Discussion 20m
      Speaker: David Rohr (CERN)

      Color code: (critical, news from this week: blue, news from last week: purple, no news: black)

      Sync reconstruction

       

      Async reconstruction

      • Need to investigate short GPU stall problem.
      • Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
      • Will tune existing 16-core settings, add a SITEARCH for 16core CPU, and 16coreCPU + generic NVIDIA / AMD GPU, like for 8 core.
      • Will retune EPN async workflow for TPC + ITS on GPU on 2025 data.

       

      GPU ROCm / compiler topics:

      • Problem with building ONNXRuntime with MigraphX support.
      • Need to find a way to build ONNXRuntime with support for CUDA and for ROCm.
      • Try to find a better solution for the problem with __device__ inline functions leaking symbols in the host code.
      • Bumped to CUDA 13.4 and ROCm 10.1, we can now bump to GCC 16.2.1
      • SLC10-FullCI running and looks healthy. We should now drop the SLC9-FullCI, and move the remaining required tests from in there to the normal SLC9-CI.

       

      TPC / GPU Processing 

      • WIP: Use alignas() or find a better solution to fix alignment of monte carlo labels: https://its.cern.ch/jira/browse/O2-5314
      • Need to check the problem with ONNX external memory allocator.
      • Next high priority topic: Improvements for cluster sharing and cluster attachment at lower TPC pad rows. PR: https://github.com/AliceO2Group/AliceO2/pull/14542
      • Check for unnecessary f64 instructions in GPU code.
      • Significant progress in TPC tracking developments. Phase 2 changes are completed, need few more days of validation and tuning, and can be merged. The extrapolation feature of Phase 3 is fully working, and will also be merged. PR: https://github.com/AliceO2Group/AliceO2/pull/15318 . For details, see S&C meeting of Monday. Will present in more details at the TPC Calib meeting next week.

       

      Other topics:

      • GCC bump - Status?

       

      EPN GPU Topics:

    • 10:20 → 10:25
      TPC ML Clustering 5m
      Speaker: Christian Sonnabend (CERN)
    • 10:25 → 10:30
      GPU Parameter Optimizations 5m
      Speaker: Gabriele Cimador (CERN, Università and INFN Torino)

      No news

    • 10:30 → 10:35
      Efficient Data Structures 5m
      Speaker: Dr Oliver Gregor Rietmann (CERN)
       

      Next Generation Triggers

      O2 CI Pipeline

      On every commit on the dev branch:

      1. Build non-deterministic and benchmark
      2. Plot the benchmarks with the different commits on the x-axis

       

      Possible implementation:

      1. Do this in the dev branch via a pipeline that dumps the benchmark results
      2. Use a manually triggered pipeline that fetches the results from Step 2 and plots them.

       

      O2 TPC Track Reconstruction

      • Working on this fork of the AliceO2 repo, with a CI pipeline:
        • Running on NGT hardware with 4 different GPUs (Nvidia and AMD)
        • Extended CI-pipline to fail if GPU.out changes
      • Implemented SoA in SectorTracker: GPUTPCBaseTrackParam, GPUTPCTrackParam, GPUTPCTracklet, GPUTPCTrack

       

      • Adressing feedback of CHEP 2026 review
      • Merged newest dev branch into my branches
      • Using newest version of dependencies from CVMFS
      • Make better use of SoA to improve performance
        • TrackletConstructor
        • TrackletSelector 
        • ExtrapolationTracking
          • Select extrapolation candidates during TrackletSelector and process only those in ExtraploationTracking
          • Storing left and right extrapolation candidates separately
          • On MI300x we see the best improvement (6% Wall time), the only one with wavefront size 64.
          • Old: middle sector waits for left and right. New: only wait for right neighbor.
          • Drawback: I had to make the data of the other sectors readable through a sectors processor.

      • Next Steps
        • Generate GPUParameters.csv file for the newest changes, maybe using a CI pipeline?
        • Schedule the two TPC rings to disjoint streams and/or use 9 streams.
    • 10:40 → 10:45
      TPC Clusterization / OpenCL / Highly Ionizing Particles 5m
      Speaker: Felix Weiglhofer (CERN)

      OpenCL

      No news.

      GPU Servers

      • CI Server: Test for deterministic mode done (https://github.com/alisw/alidist/pull/6423)
        • Added --PROCdebugDumpChecksum to compute checksum of dumped data on the fly
        • So we need to compare only hashes of dumps
        • Testdata stored on alien: /alice/cern.ch/user/a/alienci/gpuci/events
        • Also allows to reproduce test locally
        • Test works on CI machine
        • TODOs:
          • Try David's comment to also recreate remaining dump files
          • Need a way to create and pass alien tokens to the CI jobs

       

      Highly Ionizing Particles

      • Adjusted fanout behavior:
        • Old, more aggressive fanout: lost 0.8% tracks
        • +-2 timebins and pads: +0.1% tracks
        • no fanout: +0.3% tracks

       

      Tried plotting dE/dx (qTot) and rejected tracks (no filter, old filter, +- 2 fanout, no fanout):

    • 10:45 → 10:50
      ITS Tracking 5m
      Speakers: Felix Schlepper (CERN, Heidelberg University (DE)), Gabriele Cimador (CERN, Università and INFN Torino), Matteo Concas (CERN)

      Reprocessed run 572622 (LHC26ak, Pb–Pb at 23 kHz) with new GPU vertexer

      Next steps

      • Saw -1.3% yield for D0s and -0.46% yield for K0s
      • One flag removes the events: kNoSameBunchPileup

      • It rejects a collision if another reconstructed collision is assigned to the same bunch crossing

      • The new vertexer finds more low-multiplicity vertices (5–49 contributors). Some get assigned to the bunch crossing of a nearby central collision, so the central collision looks like same-bunch pile-up and is thrown away

      • Investigate this issue
      • Tune params better for UPC
      • Use GPU Tuner without RTC

       

    • 10:50 → 10:55
      System Run Coordination Topics 5m
      Speaker: Ernst Hellbar (CERN)