Alice Weekly Meeting: Software for Hardware Accelerators / PDP Run Coordination / Full System Test

Wednesday 20 Oct 2021, 11:00 → 12:20 Europe/Zurich

11:00 → 11:20 PDP Run Coordination

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

Event Display Commissioning

• ED PC delivered and working, will, bring it to the ARC today.

Problems during operation

• TPC GPU processing crashing regularly since we updated to ROCm 4.3.
  • No news, using the workaround for the pilot beam, must be fixed before data taking next year.
• EPN startup failure, nodes that timed out during SetProperty were not reported by ODC as failing --> difficult which node broke the run. Logging improved after ODC update, now reports failing collection (still not failing node), but node can be identified my matching task IDs manually. Next ODC update shall also print failing nodes.
  • One failing node identified and removed from the pool, EPN experts must have a look.
• Interface for odc plugin changed with new ODC version, causing random run startup failues since TfBuilder was still using dds plugin. Fixed by switching to odc plugin.
• Problems when writing raw data to EPNs, transferring them to EOS.
  • Some nodes have a reduced transfer rate to EOS (some high InfiniBand error counters were seen on some nodes, not yet clear if related): https://alice.its.cern.ch/jira/browse/R3C-644
  • From some nodes I saw backpressure from the TfBuilder despite no backpressure from the DPL chain. Is this because the writing to disk was too slow? https://alice.its.cern.ch/jira/browse/R3C-645

Reducing overhead from headers:

• Matthias is working on this: https://alice.its.cern.ch/jira/browse/O2-2395.
• A PR will be prepared, aim to merge it at the end of October after the pilot beam.
• Might interfer with timeframe throttling mechanism, to be checked and perhaps rebased.

Issues on EPN farm affecting PDP:

• AMD GPUs currently not working on CS8, investigating, for the moment the EPN must stay at CC8.

Issues currently lacking manpower, waiting for a volunteer:

• Tool to "own" the SHM segment and keep it allocated and registered for the GPU. Tested that this works by running a dummy workflow in the same SHM segment in parallel. Need to implement a proper tool, add the GPU registering, add the FMQ feature to reset the segment without recreating it. Proposed to Matthias as a project. Remarks: Might be necessary to clear linux cache before allocation. What do we do with DD-owned unmanaged SHM region? Related issue: what happens if we start multiple async chains in parallel --> Must also guarantee good NUMA pinning.
  • GSI group does not want to integrate it, since they say it is too ALICE specific, but they will assist. Contacted Volker about who will take care.
• For debugging, it would be convenient to have a proper tool that (using FairMQ debug mode) can list all messages currently in the SHM segments, similarly to what I had hacked together for https://alice.its.cern.ch/jira/browse/O2-2108

Workflow repository

• Waiting for AliECS to implement new fields in the GUI, Update the GUI with Vasco after discussing the shifter work with Taku. Interface to ODC to be implemented by Teo as soon as possible.
• Calibration workflow support fully available and tested with ITS.
  • Required some additional ODC updates for the odc-epn-topo tool. Were deployed yesterday.

InfoLogger messages:

• Campaign to reduce InfoLogger messages almost completed.
• ODC messages to infologger were throttled since a message flood was detected regularly. Disabled for ODC.
• ODC log messages from plugins currently do not go to the InfoLogger (in particular we do not see which nodes are used for the run). Will be fixed in the next ODC version.
• Ole will check the InfoLogger regularly and ping the responsible persons if their devices have the wrong logger severity set.
• My tool to forward all processes stderr messages with regex filter deployed on the EPNs and working, should now finally show all process crashes in InfoLogger.
• PartitionID does not show up in ODC InfoLogger messages, runNr still not, since it is not transferred from AliECS to ODC. Will be added with next Interface update.
• Run is not stopped when processes die unexpectedly. This should be the case, at least optionally during commissioning: https://alice.its.cern.ch/jira/browse/O2-2554. Will get FATAL messages in that case with next ODC version, but stopping of the run not yet possible.

Freezing of FLP software:

• New FMQ version available, but will delay deployment until after pilot beam. Will also require change in DD.
• Giulio will keep an eye that no new PRs rely on updated dependencies. Could also be hard-coded in the defaults files in alidist.

Memory monitoring:

• Added DPL metrics for available SHM memory in DPL chain: https://alice.its.cern.ch/jira/browse/R3C-638
• Johannes working on new monitoring page, that will give an overview of memory / CPU / GPU usage of all nodes, to be ready this week.
• Progress on time frame throttling mechanism to avoid getting stuck when running out of SHM, but not yet ready. https://alice.its.cern.ch/jira/browse/O2-2589
• Reactivated a node check script I was using for the HLT, to get an overview of node health status (also GPU clocks, ECC errors, IB error counters).

EOS Cleanup:

• Currently ~35 PB of commissioning data on the disk buffer, not a problem for pilot beam data taking.
• Latchezar will prepare a data base dump with all files stored on EOS not in our file catalogue. I'll create an overview from that about files / runs / sizes, and provide it to the detectors.
• Detectors shall notify us which data to keep and migrate to the catalogue by 1st of December.
• After a grace period until 1st of February, other commissioning data will be detelted.
• This was announced to the detectors by RC, together with a reminder how to access the data in the catalogue.

Having more EPNs available for processing / orchestrating detector expert access to the EPNs:

• Currently only 139 out of the 250 EPNs are available for processing. To increase the margin, could we try to move more EPNs to the online zone? https://alice.its.cern.ch/jira/browse/O2-2647
• Ole is doing a survey to get a summary for what reason detector experts need EPN access. We should close the access at some point for non-critical requests. I.e. it is not needed to access the EPN to get raw data, can be fetched from the Grid. Access to logs will soon also be obsolete, when all will go to the InfoLogger.
• At EPN group:
  • In order to not allocate nodes statically for tests (except for those with different GPU drivers), could we have a Slurm / Condor / etc. scheduler, such that experts can requests nodes?
  • I'd propose to separate the tests from the operation as much as possible. Can we use some old HLT servers, to have a separate login node for detector experts, and a set of O(5) test nodes (with the identical software as on the other EPNs). Ideally managed via Slurm / etc.
  • Detectors keep asking which EPNs to use for tests, e.g. also for data replay tests to fix crashes. Manual node allocation is impractical.

11:20 → 11:40 Full System Test

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

Full system test status

• Full system test with 2 NUMA domains not terminating correctly. DD currently forwards EOS only to one consumer: https://alice.its.cern.ch/jira/browse/O2-2375. This is also a problem for some workflows on the EPN. We should think of a general solution.
  • Feature available in FairMQ 1.43, will require small change in DD. Due to software freeze, we will not roll this out until after the pilot beam.
  • Afterwards will need change in StfBuilder / TfBuilder to keep sending EOS until all peers are disconected.
• Observed new memory leak, investigation led to 3 issues:
  • Crash in PHOS raw2cell converter: Dmitri notified, and reproducer provided. Worst we will have to assist here, but will require digging into PHOS raw data format.
  • TOF was leaking memory due to DISTSUBTIMEFRAME requested in 2 consecutive processing steps. This is a bogus topology anyway, though I am not 100% sure why this was causing the leak.
  • Found a reproducible crash in the TOF compressed-decoder at workflow termination, if the wrong topology is running. Not critical, since it is only at the termination, and the fix of the topology avoids it in any case. But we should still understand and fix the crash itself. A reproducer is available.

AMD / GPU stability issues in FST:

• Internal compiler error with -O0 is fixed.
• Minimal reproducer for internal compiler error with LOG(...) macros enabled is fixed, but full application still does not compile. Need to create a new minimal reproducer.
• Random crashes processing cosmics data with both ROCm 4.1 and 4.3, data driven race condition, not seen before. Not clear whether application bug or something else. Does not appear any more in ROCm 4.5 beta. Either it was a ROCm bug that was fixed in the meantime, or the race condition is not visible anymore due to different performance.
• Performance issue with ROCm 4.5 beta confirmed by AMD. ~10% worse processing speed, and 2.5x compile time increase. Sould be fixed before 4.5 release.
• Observed new amdgpu kernel module crash taking down the server. Happened 6 times so far. At one time it hit 3 servers in the very same second, we assume it could be related to TPC data? Or power instability? Anyway, cannot reproduce this, since the raw data during the crash is lost.
  • AMD has confirmed the bug, but they also cannot reproduce it yet.
• Observed another random server crash at application termination on the test server (kernel log does not state any module, but "Kernel NULL ptr dereferencing") Happens in "normal" application shutdown, not in the test cycle we were using before to check the stability. Must check if this is a hardware defect of the test server, or a general problem. Happens only when all 8 GPUs are used so far.
• Some GPUs don't have their clocks set to high, despite this is requested by the rocm-smi tool. Under investigation.

GPU Performance issues in FST

• One infrequent performance issue remains, single iterations on AMD GPUs can take significantly longer, have seen up to 26 seconds instead of 18 seconds. Under investigation, but no large global effect.
• Performance of 32 GB GPU with 128 orbit TF less than for the 70 orbit TF we tested in August. Results a bit fluctuating, but average is between 1600 and 1650 GPUs (compared to 1475 GPUs for 70 orbit TF). Matteo has implemented a first version of the benchmark, it is currently running on the EPN.

Important general open points:

• Multi-threaded GPU pipeline in DPL: https://alice.its.cern.ch/jira/browse/O2-1967 : Giulio will follow this up, but currently some other things still have higher priority.
• Chain getting stuck when SHM buffer runs full: on-hold: long discussion last week but no conclusion yet. All discussed solutions require knowledge how many TFs are in flight globally on a processing node, which is not yet available in DPL. https://alice.its.cern.ch/jira/browse/O2-2589.
  • Giulio working on this with high priority now, since it is also on the critical path for commissioning. Harmless parts of the work split off and already merged. PR for the "processed timeframe" metric exists, is being tested. Afterward need another PR to communicate that number to the readout-proxy.

Minor open points:

• https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
• https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
• https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
• https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
• https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
• https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
• DPL Raw Sequencer segfaults when an HBF is missing. Fixed by changing way raw-reader sends the parts, but Matthias will add a check to prevent such failures in the future.

Detector status:

• EMCAL errors (no crash, just messages, EMCAL is working on it).
• Have observed new type of EMCAL decoder errors in simulated data. Was due to different treatment of MC and real data. Already fixed.

11:40 → 12:00 Software for Hardware Accelerators

Speakers: David Rohr (CERN), Ole Schmidt (CERN)

Report on GPU memory micro benchmark progress

• Didn't have time to discuss with Matteo (sorry)

ITS GPU Tracking and Vertexing:

• Matteo will continue work on the tracking after the memory micro benchmarks are implemented.

TRD Tracking

• Working on strict matching mode (filter out ambiguous matches to obtain very clean track sample)
• David and Ole need to sit together to commission the TPC-TRD tracking on GPUs (as soon as possible...)

GPU monitoring

• Deployed by Johannes.

ANS Encoding

• Michael will implemented the proposed improvements in C++.