Alice Weekly Meeting: Software for Hardware Accelerators / PDP Run Coordination / Full System Test

Wednesday 8 Dec 2021, 11:00 → 12:20 Europe/Zurich

11:00 → 11:20 PDP Run Coordination

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

Reducing overhead from headers:

• Matthias is working on this: https://alice.its.cern.ch/jira/browse/O2-2395.
  • Most preparation work merged, can now start to really used the optimization.

Issue with start / stop / start cycle.

• DPL processes die if ECS does a cycle start / stop / start. Currently this means a new partition must be created for every run: https://alice.its.cern.ch/jira/browse/O2-2559
  • Adam reported one problem has been fixed in Monitoring.
  • Still crashing also with new monitoring version, reported on JIRA.
  • Matthias will follow up after the work with the headers is done.

Proper EOS handling in workflows (current plan):

• Created a JIRA ticket with the detils here: https://alice.its.cern.ch/jira/browse/O2-2715
• One question: how do we handle the STOP transition RUNNING --> READY. Should we do something similar?

Issues on EPN farm affecting PDP:

• AMD GPUs currently not working on CS8, investigating, for the moment the EPN must stay at CC8.

New defaults for EPN:

• EPNs should not use dataflow defaults anymore, since it creates OpenSSL problems with alien access.
• We want to use -O3 / -march for the EPN builds, thus we have create `defaults-o2-epn.sh`, which is a clone of `defaults-o2.sh` but with different CFLAGS. To be tested (on my todo-list), then we switch the nightly EPN builds to these defaults.

Issues currently lacking manpower, waiting for a volunteer:

• Tool to "own" the SHM segment and keep it allocated and registered for the GPU. Tested that this works by running a dummy workflow in the same SHM segment in parallel. Need to implement a proper tool, add the GPU registering, add the FMQ feature to reset the segment without recreating it. Proposed to Matthias as a project. Remarks: Might be necessary to clear linux cache before allocation. What do we do with DD-owned unmanaged SHM region? Related issue: what happens if we start multiple async chains in parallel --> Must also guarantee good NUMA pinning.
  • Also discussed in Pilot Beam Wrap up meeting, EPN discussing internally how to develop this. Absolutely needed for Pb-Pb run with 8 GPUs, but should tested it in pp run already obviously.
  • One problem that came up in discussions with AMD is that currently the memory registration cannot be shared between a background daemon and processes started by DDS due to different parent IDs. To be discussed how this is solved.
• For debugging, it would be convenient to have a proper tool that (using FairMQ debug mode) can list all messages currently in the SHM segments, similarly to what I had hacked together for https://alice.its.cern.ch/jira/browse/O2-2108

Workflow repository

• Merge of O2DPG and O2DataProcessing planned end of this week.

InfoLogger messages:

• Situation ok but can still be improved, particularly detector errors.
  • All detector errors that do not need action by the operations should be demoted from error to alarm.
  • Important warnings that we want to see in the InfoLogger should be promoted to alarm.
  • Severity limit for infologger will be raised to state - no more warnings shown.

Memory monitoring:

• Initial version of time frame throttling merged (https://alice.its.cern.ch/jira/browse/O2-2589)
  • Uses some ad-hoc code to set up a manual out-of-band FMQ channel.
  • Added optional free-SHM based throttling in addition.
  • Backpressure problem understood, fix in progress.
  • Giulio is first implementing a suggestion by Ruben which will make the topology more robust.
• Work on extended more elaborate version started.
  • Same functionality also needed in CTF reader, raw file reader, raw tf reader.
    • Currently async reco is extremely inefficient since it uses a conservative very long delay between time frame publishing, wastes many compute cycles. Must switch to this throttling mechanism.
  • Giulio will finalize the work on "out-of-band" channels, which will be used to transfer the metric (so the ad hoc code is not duplicated to all involved devices).
  • David will extract the throttling mechanism in a class to be reused in the other readers: Done.

EOS Cleanup:

• Got replies from all detectors but MCH. Requested migration of 4.7 PB of commissioning data (in addition to the 9 PB we have already stored in the catalogue.)
  • We can delete 2.6 PB of it in February
    • (TRD Krypton raw data, must verify before that raw-reader is OK)
    • (ITS cosmics raw data, currently need data to understand readout errors)
• Stefano and Latchezar will follow this up from now on, will not be reported here any more.

Having more EPNs available for processing / orchestrating detector expert access to the EPNs:

• Ole finished the survey for EPN access use cases.
  • Access via htcondor should be sufficient for almost all use cases.
  • Access to log files still requires direct access to online compute nodes.
    • Implementing a restricted ssh access scheme that can only do ls / cat / tail.
  • Some experts requested fast access to data while it is recorded. Should be double-checked. In principle data goes to EOS immediately.
• Set up htcondor on 3 nodes as test (login node, htcondor head node, 1 compute node).
  • Still work to be done to test with 1 / 8 core queue.
  • GPU support.

Important framework features to follow up:

• Bug: ROOT writer storing TFs to out of sync entries in the ROOT file if DPL pipelines are used.
  • Fix via completion policy in https://github.com/AliceO2Group/AliceO2/pull/7536, but needs additional framework support.
• Grafana metrics: done
  • Might want to introduce additional rate metrics that subtract the header overhead to have the pure payload.
• Logging which process processed which TF (to trace what happened when the chain got stuck): done
• Throttling of TF rate in case SHM runs full: still work in progress (see above).
• Backpressure reporting when there is a only 1 input channel: no progress.
• (Not for this week) multi-threaded pipeline: no progress.
• (Not for this week) Problem with forwarding of multi-spec output: mid-entropy encoder receives the TF twice: no progress.
• Stop entire workflow if one process segfaults / exits unexpectedly. Was in my opinion never working in the FST. According to Giulio should have worked, but is currently broken for other reasons. Need to fix current issue and then retest in FST.

11:20 → 11:40 Full System Test

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

Full system test status

• Full system test with 2 NUMA domains not terminating correctly. DD currently forwards EOS only to one consumer: https://alice.its.cern.ch/jira/browse/O2-2375. This is also a problem for some workflows on the EPN.
  • Solved with new FairMQ / DataDistribution version. Tested in Full System Test. Should be tested at P2 with 8 GPU configuration.
• Found a reproducible crash (while fixing the memory leak) in the TOF compressed-decoder at workflow termination, if the wrong topology is running. Not critical, since it is only at the termination, and the fix of the topology avoids it in any case. But we should still understand and fix the crash itself. A reproducer is available.

AMD / GPU stability issues in FST:

• ROCm 4.5 released. Currently being tested, some issues are fixed, but for now we cannot use it:
  • Performance regression fixed: 4.5 beta was ~10% slower than 4.1, 4.5 release is ~2% faster than 4.1.
  • Random crashes reproducing cosmics data: not yet clear.
  • Compile time regression partially fixed: 72 seconds with 4.5 release, 160 seconds with 4.5 beta, 52 seconds with 4.1. Reported to AMD, not sure if actual problem.
  • Internal compiler error with LOG(...) macros still exists, preparing minimal reproducer.
  • Full System Test failing dur to memory problem with ROCm 4.5. Confirmed by AMD that this is a regression with a new bug in the kernel module. Waiting for a fix to continue testing.
• Open issues not yet tested with 4.5:
  • 2 types of server crashes with kernel crash log hinting to amdgpu kernel module. Rare, not easily reproeducible. Not clear if the 2 are the same problem. AMD confirmed that at least 1 case is a problem they also see in their lab.
  • Server crash with "Kernel NULLPTR dereferencing caused by GPU monitoring via rocm-smi.
  • Some GPUs don't have their clocks set to high, despite this is requested by the rocm-smi tool. Under investigation.

GPU Performance issues in FST

• One infrequent performance issue remains, single iterations on AMD GPUs can take significantly longer, have seen up to 26 seconds instead of 18 seconds. Under investigation, but no large global effect.
• Performance of 32 GB GPU with 128 orbit TF less than for the 70 orbit TF we tested in August. Results a bit fluctuating, but average is between 1600 and 1650 GPUs (compared to 1475 GPUs for 70 orbit TF). Matteo has implemented a first version of the benchmark, it is currently running on the EPN.

Important general open points:

• Multi-threaded GPU pipeline in DPL: https://alice.its.cern.ch/jira/browse/O2-1967 : Giulio will follow this up, but currently some other things still have higher priority.

Minor open points:

• https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
• https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
• https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
• https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
• https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
• https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
• DPL Raw Sequencer segfaults when an HBF is missing. Fixed by changing way raw-reader sends the parts, but Matthias will add a check to prevent such failures in the future.

Detector status:

• EMCAL errors (no crash, just messages, EMCAL is working on it).

11:40 → 12:00 Software for Hardware Accelerators

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Report on GPU memory micro benchmark progress

• Matteo will report on the status.
• Benchmark implementation basically finished.
• Could find several conditions under which AMD memory performance decreases, but nothing fully matches  the behavior we see in O2. The only related regression is a general 6% sequential read speed decrease when using the upper and the lower 16 GB memory halfs in parallel, but cannot explain the 10% performance decrease in O2.
  • For some more details see my S&C week presentation.
  • Matteo will present a more detailed overview next week.
  • One session learned: AMD parameter tuning depends on the TF size (while for NVIDIA it does not). I.e. we should rerun the AMD parameter range scan for TFs of the full size (was so far done only for 100-collisions TFs to speed it up). And we must use separate "training" and "test" data sets.

ITS GPU Tracking and Vertexing:

• Matteo will continue work on the tracking after the memory micro benchmarks are implemented.

TRD Tracking

• Working on strict matching mode (filter out ambiguous matches to obtain very clean track sample)
• David and Ole need to sit together to commission the TPC-TRD tracking on GPUs (as soon as possible...)

GPU monitoring

• Deployed by Johannes. Currently disabled since a rocm-smi issue kill the server. Need to test if we can reenable it with ROCm 4.5.

ANS Encoding

• Michael is implementing the proposed improvements in C++.

12:00 → 12:20 GPU Memory Benchmark Report

Speaker: Matteo Concas (INFN Torino (IT))

GPU uBenchmark.pdf

Result sheets