Login

Alice Weekly Meeting: Software for Hardware Accelerators

→ Europe/Zurich
ALICE GPU Meeting
Zoom Meeting ID
61230224927
Host
David Rohr
Useful links
Join via phone
Zoom URL
    • 10:00 AM → 10:20 AM
      Discussion 20m
      Speaker: David Rohr (CERN)

      Color code: (critical, news from this week: blue, news from last week: purple, no news: black)

      Sync reconstruction

       

      Async reconstruction

      • Need to investigate short GPU stall problem.
      • Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
      • Will tune existing 16-core settings, add a SITEARCH for 16core CPU, and 16coreCPU + generic NVIDIA / AMD GPU, like for 8 core.
      • Will retune EPN async workflow for TPC + ITS on GPU on 2025 data.

       

      GPU ROCm / compiler topics:

      • Problem with building ONNXRuntime with MigraphX support.
      • Need to find a way to build ONNXRuntime with support for CUDA and for ROCm.
      • Try to find a better solution for the problem with __device__ inline functions leaking symbols in the host code.
      • Need to check ROCm 7.2 corrtecness.

       

      TPC / GPU Processing 

      • WIP: Use alignas() or find a better solution to fix alignment of monte carlo labels: https://its.cern.ch/jira/browse/O2-5314
      • Need to check the problem with ONNX external memory allocator.
      • Next high priority topic: Improvements for cluster sharing and cluster attachment at lower TPC pad rows. PR: https://github.com/AliceO2Group/AliceO2/pull/14542
      • Check for unnecessary f64 instructions in GPU code.

       

      Other topics:

      • Need to bump ONNXRuntime to 1.24, Giulio is checking, needed for ROCm 7.2 - Status?
      • GCC bump - Status?
      • Updated required number of GPUs for Run 4 (from Ernst's reference data sets):
      Interaction Rate TPC Clusters per TF Time per TF 5090 GPUs needed 5090 GPUs (with 30% margin) CPU Cores
      20 kHz 38604835 0.40351 s 142 184 7163
      30 kHz 59228746 0.59820 s 210 273 10745
      50 kHz 93076232 0.85592 s 301 391 17908

       

      EPN GPU Topics:

      • 2 nodes with MI50 and MI100 with ALMA 10.2 and newest ROCm available for validation.
    • 10:20 AM → 10:25 AM
      TPC ML Clustering 5m
      Speaker: Christian Sonnabend (CERN, Heidelberg University (DE))

      Hunting performance regression (found and fixed):

      • NN with momentum (2.96 s/TF) is basically exactly the same speed as NN without (2.92 s/TF), std. dev. usually 0.01-0.02 s/TF
      • Finds persistently 1% more tracks

       

      MC checks pending: One run failed (old O2DPG version), next one is queued at GSI

    • 10:25 AM → 10:30 AM
      GPU Parameter Optimizations 5m
      Speaker: Gabriele Cimador (CERN, Università and INFN Torino)

      TPC benchmarks across GPUs

      • Started to benchmark using parameters tuned on lhc24ar_raw
      • Had some troubles to access NGT cluster (also now)
      • Just by taking the mean time of the first event of each dataset:
      • GPU        
        A100 IR kHz Def [s] Opt [s] Gain
          20 0,867 0,823 5,04
          30 1,566 1,230 21,41
          50 2,414 2,018 16,38
        L40S        
          20 0,571 0,571 -0,03
          30 0,865 0,835 3,38
          50 1,310 1,187 9,33
        H100        
          20 0,485 0,465 4,16
          30 0,731 0,686 6,20
          50 1,062 0,985 7,26
      • Will retune based on the 20kHz dataset to see if there are improvements
    • 10:30 AM → 10:35 AM
      Efficient Data Structures 5m
      Speaker: Dr Oliver Gregor Rietmann (CERN)
       

      NextGenTrigger Task 1.7

      • Supervising a student to work an a CMS clustering algorithm.
        • We have a first standalone c++ / alpaka version.
        • Next step: Integration in the CLUEstering algorithm (CMS clustering algo)

      Implement NGT SoA Code in O2 standalone benchmark

      • Working on this fork of the AliceO2 repo, with a CI pipeline:
        • Running on NGT hardware with 4 different GPUs (Nvidia and AMD)
        • Extended CI-pipline to fail if GPU.out changes
      • Implemented SoA in:
        • SectorTracker
          • GPUTPCBaseTrackParam
          • GPUTPCTrackParam
          • GPUTPCTracklet
          • GPUTPCTrack
      • Make better use of SoA to improve performance
        • Changed a few things about GPUTPCBaseTrackParam, but performance got only worse...
      • Next Steps:
        • Generate GPU parameters .csv file for SoA and to see if the optimal parameters are different.
    • 10:40 AM → 10:45 AM
      TPC Clusterization / OpenCL / Highly Ionizing Particles 5m
      Speaker: Felix Weiglhofer (CERN)

      OpenCL

      No news.

      GPU Servers

      • CI Server: alidist recipe for a simple GPU smoke test done (https://github.com/alisw/alidist/pull/6264)
      • New issue: GPU server had issues with handling Vault secrets (will follow up with Giulio)

      Highly Ionizing Particles

      • vectorized CPU version of tail filter done: https://github.com/AliceO2Group/AliceO2/pull/15589
      • Slides for TPC meeting: https://indico.cern.ch/event/1710007/#10-saturation-tail-filter-impl
    • 10:45 AM → 10:50 AM
      ITS Tracking 5m
      Speakers: Felix Schlepper (CERN, Heidelberg University (DE)), Gabriele Cimador (CERN, Università and INFN Torino), Matteo Concas (CERN)
    • 10:50 AM → 10:55 AM
      System Run Coordination Topics 5m
      Speaker: Ernst Hellbar