Alice Weekly Meeting: Software for Hardware Accelerators / PDP-SRC

Wednesday 3 Jul 2024, 11:00 → 12:20 Europe/Zurich

11:00 → 11:20 Discussion

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

High priority Framework issues:

• Fix dropping lifetime::timeframe for good: Still pending: problem with CCDB objects getting lost by DPL leading to "Dropping lifetime::timeframe", saw at least one occation during SW validation.
  • Fixed - Did we have any issues after the fix?.
    
  • Related problem with multi-output proxy - to be checked.
    
• Start / Stop / Start: 2 problems on O2 side left:
  • • All processes are crashing randomly (usually ~2 out of >10k) when restarting. Stack trace hints to FMQ. https://its.cern.ch/jira/browse/O2-4639
      
    • TPC ITS matching QC crashing accessing CCDB objects. Not clear if same problem as above, or a problem in the task itself:
      
• Stabilize calibration / fix EoS: New scheme: https://its.cern.ch/jira/browse/O2-4308: Status?
  
• Fix problem with ccdb-populator: no idea yet - since Ole left, someone else will have to take care.

 

Other framework tickets:

• We need to make progress with these tickets at some point...
  
• https://github.com/AliceO2Group/AliceO2/pull/12976 : Better DPL backpressure reporting: Andrea confirmed the fix works.
  
• TOF problem with receiving condition in tof-compressor: https://alice.its.cern.ch/jira/browse/O2-3681
• Grafana metrics: Might want to introduce additional rate metrics that subtract the header overhead to have the pure payload: low priority.
• Backpressure reporting when there is only 1 input channel: no progress: https://alice.its.cern.ch/jira/browse/O2-4237
• Stop entire workflow if one process segfaults / exits unexpectedly. Tested again in January, still not working despite some fixes. https://alice.its.cern.ch/jira/browse/O2-2710
• https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
• https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
• https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
• https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
• https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
• https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
• https://its.cern.ch/jira/browse/O2-4759: Run getting stuck when too many TFs are in flight.
  
• https://its.cern.ch/jira/browse/O2-4234: Reduce obsolete DPL metrics
  
• https://its.cern.ch/jira/browse/O2-4860: Do not use string comparisons to derrive processor type, since DeviceSpec.name is user-defined.
  
• Found a reproducible crash (while fixing the memory leak) in the TOF compressed-decoder at workflow termination, if the wrong topology is running. Not critical, since it is only at the termination, and the fix of the topology avoids it in any case. But we should still understand and fix the crash itself. A reproducer is available.
• Support in DPL GUI to send individual START and STOP commands.
• Problem I mentioned last time with non-critical QC tasks and DPL CCDB fetcher is real. Will need some extra work to solve it. Otherwise non-critical QC tasks will stall the DPL chain when they fail.
• DPL sending SHM metrics for all devices, not only input proxy: https://alice.its.cern.ch/jira/browse/O2-4234
• Some improvements to ease debugging: https://alice.its.cern.ch/jira/browse/O2-4196 https://alice.its.cern.ch/jira/browse/O2-4195 https://alice.its.cern.ch/jira/browse/O2-4166

 

Global calibration topics:

• TPC IDC and SAC workflow issues to be reevaluated with new O2 at restart of data taking. Cannot reproduce the problems any more.

 

Async reconstruction

• Remaining oscilation problem: GPUs get sometimes stalled for a long time up to 2 minutes. Checking 2 things:
  • does the situation get better without GPU monitoring? --> Inconclusive
  • We can use increased GPU processes priority as a mitigation, but doesn't fully fix the issue.
• ḾI100 GPU stuck problem will only be addressed after AMD has fixed the operation with the latest official ROCm stack.
  
• Did some parameter tuning for async pp and PbPb workflow on EPNs:
  
  • After ITS memory improvements and sec vtx speedup, need a different config with more compute capacity for ITS (possible due to memory improvement). New config is ~2x faster for 47kHz Pb-Pb.
    
    • It appears ITS multi-threading is no longer working well. Despite the parameter being set, it basically never exceeds 100% CPU load in top. Fixed by Matteo.
      
  • Minor changes to pp workflow, particularly since 2023 data has shorter TFs, so we can have more TFs in flight.
    
    • Limiting factor for pp workflow is now the TPC time series, which is to slow and creates backpressure (costs ~20% performance on EPNs). Enabled multi-threading as recommended by Matthias - need to check if it works.
      
  • With the new settings, we can now also create benchmarks for the improvement from the TPC track model decoding (see Gabriele's report).
    

 

EPN major topics:

• Fast movement of nodes between async / online without EPN expert intervention.
  • 2 goals I would like to set for the final solution:
    • It should not be needed to stop the SLURM schedulers when moving nodes, there should be no limitation for ongoing runs at P2 and ongoing async jobs.
    • We must not lose which nodes are marked as bad while moving.
• Interface to change SHM memory sizes when no run is ongoing. Otherwise we cannot tune the workflow for both Pb-Pb and pp: https://alice.its.cern.ch/jira/browse/EPN-250
  • Lubos to provide interface to querry current EPN SHM settings - ETA July 2023, Status?
• Improve DataDistribution file replay performance, currently cannot do faster than 0.8 Hz, cannot test MI100 EPN in Pb-Pb at nominal rate, and cannot test pp workflow for 100 EPNs in FST since DD injects TFs too slowly. https://alice.its.cern.ch/jira/browse/EPN-244 NO ETA
• DataDistribution distributes data round-robin in absense of backpressure, but it would be better to do it based on buffer utilization, and give more data to MI100 nodes. Now, we are driving the MI50 nodes at 100% capacity with backpressure, and then only backpressured TFs go on MI100 nodes. This increases the memory pressure on the MI50 nodes, which is anyway a critical point. https://alice.its.cern.ch/jira/browse/EPN-397
• TfBuilders should stop in ERROR when they lose connection.
  
• Allow epn user and grid user to set nice level of processes: https://its.cern.ch/jira/browse/EPN-349
  

 

Other EPN topics:

• Check NUMA balancing after SHM allocation, sometimes nodes are unbalanced and slow: https://alice.its.cern.ch/jira/browse/EPN-245
• Fix problem with SetProperties string > 1024/1536 bytes: https://alice.its.cern.ch/jira/browse/EPN-134 and https://github.com/FairRootGroup/DDS/issues/440
• After software installation, check whether it succeeded on all online nodes (https://alice.its.cern.ch/jira/browse/EPN-155) and consolidate software deployment scripts in general.
• Improve InfoLogger messages when environment creation fails due to too few EPNs / calib nodes available, ideally report a proper error directly in the ECS GUI: https://alice.its.cern.ch/jira/browse/EPN-65
• Create user for epn2eos experts for debugging: https://alice.its.cern.ch/jira/browse/EPN-383
• EPNs sometimes get in a bad state, with CPU stuck, probably due to AMD driver. To be investigated and reported to AMD.

 

Raw decoding checks:

• Add additional check on DPL level, to make sure firstOrbit received from all detectors is identical, when creating the TimeFrame first orbit.

 

Full system test issues:

Topology generation:

• Should test to deploy topology with DPL driver, to have the remote GUI available.
  
  • DPL driver needs to implement FMQ state machine. Postponed until YETS issues solved.

 

QC / Monitoring / InfoLogger updates:

• CTF/RAW Size monitoring: status?

 

AliECS related topics:

• Extra env var field still not multi-line by default.

 

GPU ROCm / compiler topics:

• Found new HIP internal compiler error when compiling without optimization: -O0 make the compilation fail with unsupported LLVM intrinsic. Disappeared with ROCm 6.x
• Found a new miscompilation with -ffast-math enabled in looper folllowing, for now disabled -ffast-math. Seems fixed with ROCm >= 5.5.
• Must create new minimal reproducer for compile error when we enable LOG(...) functionality in the HIP code. Verified that this is not a bug in our code but internal compiler problem. AMD has a minimal reproducer.
  
• Found another compiler problem with template treatment found by Ruben. Same problem as the previous one.
  
• Debugging the calibration, debug output triggered another internal compiler error in HIP compiler. Same problem as the previous one.
  
• ROCm 6.1 with custom compiler with additional fixes - Still trying to find a standalone reproducer for the crash in the FST.
  • Started running 8 1-GPU FSTs in parallel on the nodes. In that case, usually 0-2 of them crash immediately, with the others succeeding. So it seems to be some initialization problem.
    

 

TPC GPU Processing

• Bug in TPC QC with MC embedding, TPC QC does not respect sourceID of MC labels, so confuses tracks of signal and of background events.
• New problem with bogus values in TPC fast transformation map still pending. Sergey is investigating, but waiting for input from Alex. Ruben reported that he still sees such bogus values.
  
• Status of cluster error parameterizations
  • No progress yet on newly requested debug streamers.
    
  • Waiting for TPC to check PR with full cluster errors during seeding.
• Problem with constraining TPC tracks to TPC volume. Fixed - taking into account  TPCFastTransform::getT0
  
• TPC reported a problem with laser runs. In case of bad data (tpc data outside of the triggered drift time), GPUs sometimes can get stuck, so apparently the skipping of bad data is not fully working. Recorded some laser raw data to check.
  
• Added an option to print all TPC configKeyValues to log file, printing only for the first pipeline instance.
  

 

TPC processing performance regression:

• O2/dev:
  • • Total time: 4.695s, Track Fit Time 1.147s, Seeding Time 1.241s
  • O2/dev with commit from 4.3. reverted:
    • Total time 4.351s, Track Fit Time 1.089, Seeding Time 1.008s
  • For reference: before introduction of the V-Shape map:
    • Total time 3.8421s (didn't measure individual times)
  • O2/dev with scaling factors hard-coded to 0 (essentially using one single transformation map without any scaling):
    • Total time 3.093 Track Fit Time 0.682s Seeding Time 0.429s
• Proposed 3 ideas to speed up the map access:
  1. We merge the maps on-the-fly to one combined map, and query only one map.
  2. We could add plenty of #ifdef in the code, to make sure that for online purposes all the code for the non-static map is not seen.
  3. We could try to optimize the code to make it easier for the compiler.
• Outcome of meeting with TPC:
  • Sergey will implement a new fully flat map, and multiple existing maps will be merged into the new one.
    • This will give optimal performance for both sync and async.
    • It will also remove the memory overhead of having multiple maps in memory per process, since then we can use the flat maps from the SHM.
      
    • Timescale is ~September.
      
  • Meanwhile, I will add the possibility to remove the M/V-shape code with run time compilation, which should restore the performance from last November at P2 if we use RTC.
    

 

General GPU Processing

• Pending problems with using GPU RTC at P2:
  • /tmp is inside the slurm container and wiped afterwards. Fixed by using /var/tmp.
  • RTC is started from one of the GPU processes, which has a NUMA pinning to one NUMA domain, thus it uses only half of the CPU cores. Need to extend the CPU pinning for the compilation subprocesses.
  • RTC compiles for the architecturs of the original build, which is currently MI50/MI100, i.e. all nodes compile twice, which takes extra time. Need to add an option to select an architecture, and the topology generation must put in the setting for MI50 / MI100 architectures.
  • AMD compiler leaves stale temp folders, can avoid this by setting $TMPDIR.
  • RTC compilation fails in an online run since headers (e.g. <cmath>) are not found - Not understood yet.

11:25 → 11:30 TPC ML Clustering

Speaker: Christian Sonnabend (CERN, Heidelberg University (DE))

Framework implementation

• ONNXRuntime ultimately failed with compiler directives (asm volatile("s_mov_b32 m0, %0; \n\t") -> Chat with Giulio and Sergio: This is probably due to some incompatibility with our gcc and might only be fixable with gcc 14
• Trying new versions of PyTorch again
• Nicodemos is fully up to speed and trying new potential framework candidates: Tensorflow lite -> Light inference framework of TF made for deployment on mobile devices and various accelerators

 

NN implementation in O2

• NN's are working in the GPU clusterization code, but some issues still to be fixed (e.g. tpc-native-clusters.root is not being published correctly, to be investigated)
• Added some options (like model paths, thread settings, etc.) to GPUSettingsList.h which makes the workflow more configurable
• Built a track matcher macro which compares two tracks and their associated clusters to compare if they can be counted as the same or not (here it woud be good to sit together and decide on some criteria) 

11:30 → 11:35 ITS Tracking

Speaker: Matteo Concas (CERN)

• WIP: Understand LHC24aj,LHC24al weird track length distribution.
  • DeltaRof in tracking is not enabled in LHC24aj, yet the discrepancy is there.
• WIP: Seeding vertexer with deltaRof (on hold).
• WIP: GPU Vertexer using external allocator (on hold).

11:35 → 11:55 TPC Track Model Decoding on GPU

Speaker: Gabriele Cimador (Universita e INFN Trieste (IT))

Async reco workflow (after tuning) before vs after GPU decoding

s/TF	original CPU decoding	new GPU decoding
2023 real pp data	4.751 ± 0.038	4.617 ± 0.022
2023 real Pb-Pb data	14.353 ± 0.165	14.151 ± 0.082

Aync reco faster:

• By 2.8% for pp
• By 1.4% for Pb-Pb

CPU time of o2-gpu-reco-workflow process reduced by (relative to total cpu time):

• 5.1% for pp
• 1.6 % for Pb-Pb