Alice Weekly Meeting: Software for Hardware Accelerators / PDP Run Coordination / Full System Test

→ Europe/Zurich
Zoom Meeting ID
61230224927
Host
David Rohr
Useful links
Join via phone
Zoom URL
    • 11:00 → 11:20
      PDP Run Coordination 20m
      Speakers: David Rohr (CERN), Ole Schmidt (CERN)

      Color code: (news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

      Event Display Commissioning

      • Work ongoing to fix crash with integrated GPU.
      • Possible PC model as replacement for ARC ED PC selected.

      Problems during operation

      • Improvements for memory registration callbacks merged, in FST callbacks are coming immediately. Still need to implement waiting for all callbacks before going to READY state. Implemented by Giulio, tested and merged.

      Issues on EPN farm affecting PDP:

      • AMD GPUs currently not working on CS8, investigating, for the moment the EPN must stay at CC8.

      Issues currently lacking manpower, waiting for a volunteer:

      • Tool to "own" the SHM segment and keep it allocated and registered for the GPU. Tested that this works by running a dummy workflow in the same SHM segment in parallel. Need to implement a proper tool, add the GPU registering, add the FMQ feature to reset the segment without recreating it. Proposed to Matthias as a project. Remarks: Might be necessary to clear linux cache before allocation. What do we do with DD-owned unmanaged SHM region? Related issue: what happens if we start multiple async chains in parallel --> Must also guarantee good NUMA pinning.
      • For debugging, it would be convenient to have a proper tool that (using FairMQ debug mode) can list all messages currently in the SHM segments, similarly to what I had hacked together for https://alice.its.cern.ch/jira/browse/O2-2108

      Workflow repository

      • PDP Workflow repository used in first test in global run with TPC today to generate workflow, tests with more detectors will follow this week.
      • Waiting for AliECS to implement new fields in the GUI.
      • Need new ODC version before we can make the O2 version selectable (currently fixed to the latest)
    • 11:20 → 11:40
      Full System Test 20m
      Speakers: David Rohr (CERN), Ole Schmidt (CERN)

      Color code: (news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

      Full system test status

      • Waiting for memory registration callbacks implemented by Giulio.
      • Full system test with 2 NUMA domains not terminating correctly. DD currently forwards EOS only to one consumer: https://alice.its.cern.ch/jira/browse/O2-2375. This is also a problem for some workflows on the EPN. We should think of a general solution. No news, to be discussed with Giulio.
      • Applied some fixes such that the full system test can run with data files in an external instead of the local directory.
      • Full system test workflow now part of the PDP workflow repository as first example, creating DDS topology for full system test workflow using the parser already works.
      • Full system test has been tuned also for async processing with CPU and GPU.

      AMD / GPU stability issues in FST:

      • Compiler fails with internal error when optimization is disabled (-O0): Shall be fixed in ROCm 4.4. Waiting to receive a preliminary patch to confirm it is fixed.
      • Reported new internal compiler error when we enable log messages in GPU management code. Shall be fixed in ROCm 4.4.Waiting to receive a preliminary patch to confirm it is fixed.
      • Confirmed that the crashes we are seeing are caused by a regression in the GPU kernel module. Waiting for a fix.

      GPU Performance issues in FST

      • One infrequent performance issue remains, single iterations on AMD GPUs can take significantly longer, have seen up to 26 seconds instead of 18 seconds. Under investigation, but no large global effect.
      • Performance of 32 GB GPU with 128 orbit TF less than for the 70 orbit TF we tested in August. Results a bit fluctuating, but average is between 1600 and 1650 GPUs (compared to 1475 GPUs for 70 orbit TF). Matteo has implemented a first version of the benchmark, it is currently running on the EPN.
      • New 6% performance regression with ROCm 4.3.

      Important general open points:

      • Multi-threaded GPU pipeline in DPL: https://alice.its.cern.ch/jira/browse/O2-1967 : Partially merged, disabled at compile time, will follow-up when Giulio returns.
      • https://alice.its.cern.ch/jira/browse/QC-569 : Processors should not be serialized when they access the same input data. (reported by Jens for QC, but relevant for FST.) . New FairMQ version will be available in few days with the required features, afterwards to be implemented in DPL.
        • This feature might have some interplay with the "optional" type of the raw data messages we use in order to handle missing detectors. Might need some additional work in DPL.
      • Chain getting stuck when SHM buffer runs full: on-hold: long discussion last week but no conclusion yet. All discussed solutions require knowledge how many TFs are in flight globally on a processing node, which is not yet available in DPL. --> Implement this first, then see further.

      Minor open points:

      • https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
      • https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
      • https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
      • https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
      • https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
      • https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
      • DPL Raw Sequencer segfaults when an HBF is missing due to incorrect assumption that all pages are 8 kb except for the last (while this assumption is only true within an HBF not globally). Workaround should exist in raw-reader to send the HBFs as multiple parts, to be tested.

      Detector status:

      • EMCAL errors (no crash, just messages, EMCAL is working on it).
      • V0 still lacking reconstruction
    • 11:40 → 12:00
      Software for Hardware Accelerators 20m
      Speakers: David Rohr (CERN), Ole Schmidt (CERN)

      Report on ANS encoding progress

      • preparing a presentation with the results, to be shown in this meeting
      • memory consumption with ALICE simulated data increased wrt random data
      • with optimizations of the renorming the memory consumption, especially for the decoding, which is more demanding than the encoding, can be improved
      • focus should be computing speed and compression factor, which is already very good for TPC
      • Michael will send a written report due to connectivity issues

      Report on GPU memory micro benchmark progress

      • still preparing a presentation with the results, to be shown in this meeting
      • fixed a CMake issue in the benchmark that was breaking the CI.

      TRD Tracking

      • Working on strict matching mode (filter out ambiguous matches to obtain very clean track sample)
      • David and Ole need to sit together to commission the TPC-TRD tracking on GPUs (after the vacation)
      • Ole is working on the TRD refit this week.

      GPU monitoring

      • Johannes will follow this up with Alexander.