Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

Reducing overhead from headers:

• Gvozden opened PR to integrate this in DD today. Will run FST for it later today?
• FST test with new reduced-header scheme crashes.
• New dev branch of DD with option to use old/new header scheme. Tested in FST. Works with old scheme, crashes with new one. To be merged, since it requires necessary changes for other developments. Will follow up the crash with the new scheme afterwards.

Issue with start / stop / start cycle.

• DPL processes die if ECS does a cycle start / stop / start. Currently this means a new partition must be created for every run: https://alice.its.cern.ch/jira/browse/O2-2559
  • Didn't follow it up. One JIRA ticket was closed. Is this working at P2 now?

Proper EOS handling in workflows (current plan):

• PR created and merged by Giulio, tested in FST that there are no side-effects if not used.
• Needed additional features in the Debug GUI to test the EOS handling. PR there for it by Giulio, but not yet tested.

GPU ROCm Issues to be followed up by EPN:

• EPN team will setup a test script for future validation of ROCm relases
• Check if ROCm 4.5 fixed the server crashes we had when enabling GPU monitoring
• Check if ROCm 4.5 fixes the issues that GPU clock speeds are sometimes set to low
• Create new minimal reproducer for compile error when we enable LOG(...) functionality in the HIP code. Check whether this is a bug in our code or in ROCm. Lubos will work on this.

Problem with EPN OS and GPUs:

• AMD GPUs currently not working on CS8, investigating, for the moment the EPN must stay at CC8.
• ROCm 4.5 fails to register memory when we switch to CentOS 8.5 (8.5 is the final CC8 version, since CC8 is no EOL. If we want to stick to CC8 on the EPNs for some time, perhaps it would make sense to install this latest version now).
• Inquired with AMD about future support of RHEL clones, waiting for reply.

New defaults for EPN:

• Deployed and running on the EPNs.
• Unfortunately, cannot run on old HLT nodes since no AVX 2. Deploying normal O2 build on dev nodes.
  • Asked Johannes to adapt the deployment script so that we can define different versions for prod nodes and dev nodes. (Currently, wrong version is installed on the dev node, and must be removed and replaced manually).

Full system test issues:

• Created a raw file dataset for full scale full system test from readout yesterday. Waiting for reply from Sylvain whether this works for him:

Issues currently lacking manpower, waiting for a volunteer:

• For debugging, it would be convenient to have a proper tool that (using FairMQ debug mode) can list all messages currently in the SHM segments, similarly to what I had hacked together for https://alice.its.cern.ch/jira/browse/O2-2108
• Redo / Improve the parameter range scan for tuning GPU parameters. In particular, on the AMD GPUs, since they seem to be affected much more by memory sizes, we have to use test time frames of the correct size, and we have to separate training and test data sets.

InfoLogger messages:

• Finalized cleanup campaign.
  • Went through all errors / warnings during pilot beam, and adjusted severities, such that detector issues show as alarm.
  • Minimum severity raised to "important", i.e. "warnings" are no longer shown in InfoLogger, only in local log files on the nodes.
  • Collection of important alarm/error messages will be added to shifter/oncall instruction by Ole, together with some hint what is the background / whom to call.

Time frame throttling:

• Not working together with QC (thus unusable for sync reco).
• Throttling mechanism already extracted to dedicated class, to be used in different places.
• In order to use it for other cases but the raw-proxy, the ad-hoc setup of the out-of-band FMQ channel must be replaced by a generic out-of-band channel DPL feature.

Speeding up startup of workflow on the EPN:

• Sorted out some additional issues / hangs with usage of SHM Manager tool. Should not support also the case of crashed processes.
  • Tested yesterday in several iterations running many 8 GPU FSTs in the same segment and killing them with -9. Seems stable, but obviously that does not cover all cases.
• AliECS GUI changes to configure the segment ID for the workflow in case the tool is used merged. (Disabled by default, but can be enabled in the expert panel, and then we can test it in a real partition with the tool). Deployed with FLPSuite update on Monday.
• All done from the PDP side, startup time in standalone measurement: 18 seconds.
• Next steps:
  • Get R3C-646 and R3C-696 closed.
    • Deploy the tool on the EPN.
    • Add automatic cleanup in the EPN state machine.
    • Wait for new DDS version to support DPL config integrated in DDS XML topology file.
  • Remeasure in a real partition controlled by AliECS, compare the time, and see where we have delays and loose time since things don't run in parallel.

• New feature: we must not create the DPL workflow JSON for each individual process (this means we start O(1000) processes each start of run just to create the JSONs. Discussing with the DDS team here: https://alice.its.cern.ch/jira/browse/R3C-696. Anar is implementing a config file feature in DDS: https://github.com/FairRootGroup/DDS/issues/406

EPN Scheduler

• EPN access no restricted to only the login node, all further access via slurm (except for some selected people, who need e.g. to attach a debugger to sync processing).
• Set up a group to administrate who can submit to the prod partition (currently Matteo for GPU tests).
• Fixed issue with GPU access permissions (thx to Matteo for reporting).
• Next step: Add VObox for GRID

Important framework features to follow up:

• Multi-threaded GPU pipeline in DPL: https://alice.its.cern.ch/jira/browse/O2-1967 : Giulio will follow this up, but currently some other things still have higher priority.
• Bug: ROOT writer storing TFs to out of sync entries in the ROOT file if DPL pipelines are used.
  • Fix via completion policy in https://github.com/AliceO2Group/AliceO2/pull/7536, but needs additional framework support.
• Suppress default options when generating DDS command lines: https://alice.its.cern.ch/jira/browse/O2-2736 - will drop this, not needed once we cache the DPL JSON topology.
• Grafana metrics: Might want to introduce additional rate metrics that subtract the header overhead to have the pure payload: low priority.
• Backpressure reporting when there is a only 1 input channel: no progress.
• (Not for this week) multi-threaded pipeline: no progress.
• (Not for this week) Problem with forwarding of multi-spec output: mid-entropy encoder receives the TF twice: no progress.
• Stop entire workflow if one process segfaults / exits unexpectedly. Tested again in January, still not working despite some fixes. https://alice.its.cern.ch/jira/browse/O2-2710
• Output proxy spinning at 100%: no progress.

EPN O2 Software deployment:

• EPN O2 software updated twice: 1st to be compatible to FLP QC; 2nd for the workaround for TPC GPU issue.
• Encountered several issues during deployment last week:

  • QC incompatible with the version on the FLPs.

  • Build failed due to old fmt in flpsuite alidist branch used.

  • TBB ROOT include path missing in one build.

  • O2-epn defaults failed with O2Physics (due to old O2Physics being used), but O2Physics should not be built for the EPN anyway.

  • But in RPM creation leading to file collisions, so only one version could be installed at the same time.

• Lots of time spend on updating / testing / fixing this, hopefully the next updates will be cleaner.

AMD / GPU stability issues in FST:

• New compiler bug in AMD hipcc encountered, recent changes in our code (unrelated to TPC) together with high optimization level (-O3) and function calls enabled (recommended by AMD for our code) leads to miscompilation of TPC looper following code.
  • Reported to AMD, Lubos is preparing a test node for them to reproduce it.
  • In the meantime, as workaround the optimization level is reduced to -O2, at moderate performance penalty.
• ROCm 4.5 released. Result of tests of open issues:
  • Random server crashes with error reported in amdgpu kernel module in dmesg: Not fully fixed, had at least one crash now, but with different dmesg message compared to before. Not clear if same issue or something different (or a hardware error, happened only once, but have only one test node so far).
  • Random crash with noisy pp data: Disappeared with ROCm 4.5. Cannot reproduce it anymore. Was never understood. Hopefully it was a bug in previous ROCm that was fixed by chance in 4.5. Closed now.
  • Random crash processing Pb-Pb data: still there, but happens similarly as in ROCm 4.3, thus no regression in 4.5. Need to debug further what exactly happens and then report to AMD.
  • Error with memory registration: fixed.

GPU Performance issues in FST

• One infrequent performance issue remains, single iterations on AMD GPUs can take significantly longer, have seen up to 26 seconds instead of 18 seconds. Under investigation, but no large global effect.

Minor open points:

• https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
• https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
• https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
• https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
• https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
• https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
• DPL Raw Sequencer segfaults when an HBF is missing. Fixed by changing way raw-reader sends the parts, but Matthias will add a check to prevent such failures in the future.
• Found a reproducible crash (while fixing the memory leak) in the TOF compressed-decoder at workflow termination, if the wrong topology is running. Not critical, since it is only at the termination, and the fix of the topology avoids it in any case. But we should still understand and fix the crash itself. A reproducer is available.

Workflow repository:

• Next FLPSuite update will provide more configuration options for PDP expert panel:
  • O2DPG (workflow), O2PDPSuite (software), QC JSON (qc config) versions can be set individually. By default, will be the string `default`, in which case default versions are fetched from consul, that can be configured by RC.
    • With this, we can deploy new O2 versions on the EPNs in parallel, test them, and upgrade / downgrade can be handled by RC by just changing the setting in consul.
    • QC JSONs are still not versioned in consul, i.e. whenever we get a new version from AliECS, we fetch the JSONs, and store them in the workflow cache using the version provided by AliECS. This means, when detectors update the JSON files in consul, they need to bump this version manually for now (until FLP implements the versioning in consul).
      • Currently, JSONs are still not fetched from consul, due to inconsistent naming scheme used, which is currently cleaned up by Barth.
  • Supports setting shmid for shm management tool (see above).
  • Can set fraction of raw data to be stored, will be forwarded to DD (not yet implemented in DD).
  • Some cleanup.
• New GUI merged and working, spotted one bug in AliECS core, which does not transfer correct O2DPG version. Have a workaround for now. Contacted Teo to fix it.

Restarting oncalls:

• Since MW1 mid of Januarry, daily meetings have restarted, and we give support at working hours.
• Full oncalls will restart on 21st of February - we should get the booking started.
• Some of the documentation needs updating, particularly the software deployment parts.
• RC requests from us regular O2 software updates on the EPNs on Monday in parallel to the FLPSuite update. This should become oncall duty.
• Ole and David will sit together this week and go through the material and update it.
• Planning to scheduler another oncall training when the oncalls restart.

Detector status:

• EMCAL errors (no crash, just messages, EMCAL is working on it).