Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

General:

• Problem on MI100 / MI210 not solved by new ROCm beta version. AMD can reproduce the problem and is investigating.
• Locally tested OpenCL compilation with Clang 14 bumping –cl-std from clc++ (OpenCL 2.0) to CLC++2021 (OpenCL 3.0) and using clang-internal SPIR-V backend. Arrow bump to 8.0 done, which was prerequesite.

ITS GPU Tracking and Vertexing:

• Matteo will spend 1 week working on multi-threading of ITS vertexing, then go back to GPU ITS tracking.

TPC GPU Processing

• Found bug in CPU version of GPU TPC ZS v4 (DLBZS) decoder, reported to Felix.
• Investigation “random crashes” in TPC tracking: Had a look at all GPU memory fault error messages in the last 60 days: have to distinguish 2 cases:
  • Correlated crashes: gpu-reconstruction crashes on many nodes at more or less the same time. All of the cases are most certainly caused by corrupt raw data. In each case there are plenty of warnings for corrupt TPC meta data at the same time.
    • Note, that
      • In order to recover half-broken time frames, if possible also TFs with broken meta info is processed by the GPU.
      • Not all cases of broken meta information is detected ahead of time, i.e. meta-data corruption can lead to a GPU crash without prior warning message.
    • Several of these cases were also TPC standalone tests and not global runs.
  • Single crashes, where just one gpu-reconstruction process segfaults (on a longer timescale of at least one hour). Much more difficult to investigate:
    • Often accompanied by a warning for corrupt meta data by that process before, so same as above but only a single TF is corrupted.
    • Can be caused by bad gpu-memory, if it happens on the same gpu many times, particularly when it happens in the FST where the TF is known to be good. One has to collect statistics which GPUs fail often and replace them. But this has become pretty rare right now (e.g. epn199).
      • Started to create some statistics and discussing with Dirk from EPN.
    • There are several cases of only a single crash in a run, on a GPU which never crashed again. Not clear what is the cause.
      • Unlikely that the GPU is broken.
      • Likely that it is corrupt TPC metadata, since the checks that show error messages cannot detect all kinds of corruption.
      • Could of course also be a race-condition, or other bug in the code. But cannot reproduce the problem, we store only 1 permille of raw data (and none for the TPC standalone test), and we don’t have the raw files which crashed.

Unfortunately, cannot do anything more right now. Should fix the metadata corruption first, and replace GPUs that fail regularly. Only afterwards I would try to check for race-conditions / software bugs.

TRD Tracking

ANS Encoding

• Waiting for PR with AVX-accelerated ANS encoding