Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    Alice Weekly Meeting: Software for Hardware Accelerators / PDP-SRC

    → Europe/Zurich
    ALICE GPU Meeting
    Zoom Meeting ID
    61230224927
    Host
    David Rohr
    Useful links
    Join via phone
    Zoom URL
      • 11:00 → 11:20
        Discussion 20m
        Speakers: David Rohr (CERN), Ole Schmidt (CERN)

        Color code: (critical, news during the meeting: green, news from this week: blue, news from last week: purple, no news: black)

        High priority framework topics:

        Other framework tickets:

        • TOF problem with receiving condition in tof-compressor: https://alice.its.cern.ch/jira/browse/O2-3681 - Status?
        • Grafana metrics: Might want to introduce additional rate metrics that subtract the header overhead to have the pure payload: low priority.
        • Backpressure reporting when there is only 1 input channel: no progress.
        • Stop entire workflow if one process segfaults / exits unexpectedly. Tested again in January, still not working despite some fixes. https://alice.its.cern.ch/jira/browse/O2-2710
        • https://alice.its.cern.ch/jira/browse/O2-1900 : FIX in PR, but has side effects which must also be fixed.
        • https://alice.its.cern.ch/jira/browse/O2-2213 : Cannot override debug severity for tpc-tracker
        • https://alice.its.cern.ch/jira/browse/O2-2209 : Improve DebugGUI information
        • https://alice.its.cern.ch/jira/browse/O2-2140 : Better error message (or a message at all) when input missing
        • https://alice.its.cern.ch/jira/browse/O2-2361 : Problem with 2 devices of the same name
        • https://alice.its.cern.ch/jira/browse/O2-2300 : Usage of valgrind in external terminal: The testcase is currently causing a segfault, which is an unrelated problem and must be fixed first. Reproduced and investigated by Giulio.
        • Found a reproducible crash (while fixing the memory leak) in the TOF compressed-decoder at workflow termination, if the wrong topology is running. Not critical, since it is only at the termination, and the fix of the topology avoids it in any case. But we should still understand and fix the crash itself. A reproducer is available.
        • Support in DPL GUI to send individual START and STOP commands.
        • Problem I mentioned last time with non-critical QC tasks and DPL CCDB fetcher is real. Will need some extra work to solve it. Otherwise non-critical QC tasks will stall the DPL chain when they fail.

        Global calibration topics:

        • TPC calib problem: Need to repeat the test with latest fixed, but as we still have problems in all online runs, I think it will require more work..
        • TPC had problems with FLP processing stuck in IDC workflow, under investigation, but not really reproducible.
        • Even with latest changes, difficult to ensure guaranteed calibration finalization at end of global run (as discussed with Ruben yesterday).
          • After discussion with Peter, Giulio this morning: We should push for 2 additional states in the state machine at the end of run between RUNNING and READY:
            • DRAIN: For all but O2, the transision RUNNING --> FINALIZE is identical to what we do in STOP: RUNNING --> READY at the moment. I.e. no more data will come in then.
              • O2 could finalize the current TF processing with some time out, where it stops processing incoming data, and at EndOfStream trigger the calibration postprocessing.
            • FINALIZE: No more data is guaranteed to come in, but the calibration could still be running. So we leave FMQ channels open, and have a time out to finalize the calibration. If input proxies have not yet received the EndOfStream, they will inject it to trigger the final calibration.
          • This would require changes in O2, DD, ECS, FMQ, but all changes except for in O2 should be trivial, since all other components would not do anything in these states.
          • No progress...
        • Implemented TPC current integreation for FIT detectors and TOF.

        Async reconstruction

        • Remaining oscilation problem: GPUs get sometimes stalled for a long time up to 2 minutes.
          • Checking 2 things: does the situation get better without GPU monitoring? --> Inconclusive
          • We can use increased GPU processes priority as a mitigation, but doesn't fully fix the issue.
        •  

        EPN major topics:

        • Fast movement of nodes between async / online without EPN expert intervention.
        • Interface to change SHM memory sizes when no run is ongoing. Otherwise we cannot tune the workflow for both Pb-Pb and pp: https://alice.its.cern.ch/jira/browse/EPN-250
        • Improve DataDistribution file replay performance, currently cannot do faster than 0.8 Hz, cannot test MI100 EPN in Pb-Pb at nominal rate, and cannot test pp workflow for 100 EPNs in FST since DD injects TFs too slowly. https://alice.its.cern.ch/jira/browse/EPN-244 NO ETA
        • Need DDS/ODC feature to deploy different topologies on EPNs with MI50 and with MI100. ETA End of March + ~2 weeks - Status?
        • Go to error state if a critical task (e.g. calib task) fails (taking nmin into accound). But currently we do not see failing calibration tasks at all, except for a message in infologger. ODC should go to error, and then ECS should stop the run automatically. Also when n < nmin. ETA End of March - Status?
        • Lubos to provide interface to querry current EPN SHM settings - Status?
        • DataDistribution and ODC InfoLogger messages not shown at shifter console since they were at level trace. Discused with Sylvain, and implemented the same severity --> level translation as in O2. Merged in both repositories. DD already updated, ODC still not updated.

        Other EPN topics:

        • Check NUMA balancing after SHM allocation, sometimes nodes are unbalanced and slow: https://alice.its.cern.ch/jira/browse/EPN-245
        • Fix problem with SetProperties string > 1024/1536 bytes: https://alice.its.cern.ch/jira/browse/EPN-134 and https://github.com/FairRootGroup/DDS/issues/440
        • After software installation, check whether it succeeded on all online nodes (https://alice.its.cern.ch/jira/browse/EPN-155) and consolidate software deployment scripts in general.
        • Improve InfoLogger messages when environment creation fails due to too few EPNs / calib nodes available, ideally report a proper error directly in the ECS GUI: https://alice.its.cern.ch/jira/browse/EPN-65
        • If DD connection of a node fails, the node should be taken out and count against nmin, otherwise it can make the false impression that the processing on the other nodes is too slow.
          • Contributed an experimental PR that shuts down TfBuilder in this case. Need to check online if this can break something else (would anyway only happen in this special error case). Inconclusive so far: Despite trying for one night, couldn't reproduce the issue. Proposal is to just use it by default, and fall back in case of problems.

        EPN farm upgrade:

        • New AMD ROCm version 5.5 available:
          • DMA problem with MI100 and function calls enabled doesn't appear any more. Not clear if fixed or just disappeared by chance.
          • All other combinations, MI50, and MI100 without function calls do not work. Either crash or get stuck. AMD is invesigation and has a reproducer.
            • AMD has requested if we can provide a standalone reproducer for this. Cheking... but current standalone event dump is broken.

        TPC Raw decoding checks:

        • TPC raw decoding yields incorrect results when first TfOrbit is wrong (was reported by TfBuilder, but not spotted by shift crew since messages were trace level, which is fixed already).
        • Adding additional check in TPC raw decoding for sane first Orbits.
        • Will also add a check on DPL level, to make sure firstOrbit received from all detectors is identical, when creating the TimeFrame first orbit.
        • Since we had repeatedly situations where we have corrupt TPC raw data online but cannot reproduce, and DataDistribution provides no possibility to dump TFs after processing (currently we dump 1 permille, so chance to dump the corrupt TF is pretty low), will implement TPC TF dump in the TPC reconstruction, so we get that data (similarly to what ITS does).
        • Discussed with TPC and QC experts how to export raw decoding error counters from the GPU reco to QC, to visualize raw data errors in geometrical plot and create statistics.
        • Implemented general RDH size and offset checks wrt raw data buffer in DPLRawParser (will be active for all detectors but TPC and TRD). Will also solve problem of incomplete HBFs: https://alice.its.cern.ch/jira/browse/O2-3200. Required change in QC, with new QC tag released yesterday.

        Full system test issues:

        Topology generation:

        • Ole is investigating to use set -u or set -e to catch more errors, but has some drawbacks. Current plan is to use -u, but not -e. To be merged when Ole is back from vacation in 2 weeks. Merged
        • Should test to deploy topology with DPL driver, to have the remote GUI available.

        Software deployment at P2.

        • New O2PSPSuite from 26th deployed yesterday. All the raw data improvements are not yet in.
        • Have new Pb-Pb dataset for SYNTHETIC runs. More realistic TPC data sizes but still not fully realistic. Now simulating pp and then will deploy at P2.
          • Afterwards should check number of clusters and digits in more detail with more statistics, and discuss with TPC experts.
          • Found several bugs in some digitizers, which broke with longer simulation window. All fixed by Sandro.
          • New SYNTHETIC PbPb and pp datasets were provided to FLPs and distributed to servers by Sylvain.

        QC / Monitoring / InfoLogger updates:

        • TPC has opened first PR for monitoring of cluster rejection in QC. Trending for TPC CTFs is work in progress. Ole will join from our side, and plan is to extend this to all detectors, and to include also trending for raw data sizes.

        CCDB topics.

        AliECS related topics:

        • ECS should replace newlines in extra env field by spaces, but doesn't. Filed a bug report in JIRA: newline to space conversion implemented, but field is not yet multi-line by default.

        GPU ROCm / compiler topics:

        • Found new HIP internal compiler error when compiling without optimization: -O0 make the compilation fail with unsupported LLVM intrinsic. Reported to AMD.
        • Found a new miscompilation with -ffast-math enabled in looper folllowing, for now disabled -ffast-math.
        • Must create new minimal reproducer for compile error when we enable LOG(...) functionality in the HIP code. Check whether this is a bug in our code or in ROCm. Lubos will work on this.
        • Found another compiler problem with template treatment found by Ruben. Have a workaround for now. Need to create a minimal reproducer and file a bug report.
        • Debugging the calibration, debug output triggered another internal compiler error in HIP compiler. No problem for now since it happened only with temporary debug code. But should still report it to AMD to fix it.

        TPC GPU Processing

        • Random GPU crashes under investigation.
        • Sergey provided fix for inverse transformation map, but not yet merged in O2. Sergey's fix cures the problems I had with the inverse transformation, so merged in O2. 
        • Bug in TPC QC with MC embedding, TPC QC does not respect sourceID of MC labels, so confuses tracks of signal and of background events.
        • Bug in TPC Clusterizer in multi-threaded CPU DLBZS decoding, Fixed by Felix.
        • 2 Minor bugs in TPC clusterization fixed.
        • Added support for shifting TPC clusters in time (requested by Ruben) - This had introduced a bug in clusterization, which was fixed.
        • Bug in TPC processing with distortion correction with scaling - under investigation. No news...

        ANS Encoding

        • More work on the PR for AVX ANS encoding was ongoing. Passes the CI finally. Needs minor additional changes, but converging.

        Issues currently lacking manpower, waiting for a volunteer:

        • For debugging, it would be convenient to have a proper tool that (using FairMQ debug mode) can list all messages currently in the SHM segments, similarly to what I had hacked together for https://alice.its.cern.ch/jira/browse/O2-2108
        • Redo / Improve the parameter range scan for tuning GPU parameters. In particular, on the AMD GPUs, since they seem to be affected much more by memory sizes, we have to use test time frames of the correct size, and we have to separate training and test data sets.
      • 11:20 → 11:25
        TRD Tracking 5m
        Speaker: Ole Schmidt (CERN)
      • 11:25 → 11:30
        TPC ML Clustering 5m
        Speaker: Christian Sonnabend (Heidelberg University (DE))
      • 11:30 → 11:35
        ITS Tracking 5m
        Speaker: Matteo Concas (CERN)
        • Currently: Max is trying to optimise the way we do the last part of the tracking, so to reduce the time spent on that part:
          • If this goes through, it will change the algorithm's structure (Cellular automaton and KF will not be fully separated).
        • This development goes in parallel with GPU porting.
          • I'll leave the "road finding" on the CPU and move to resume the parallel track fitting on GPU, as this is the most time-consuming part.
          • If Max's change goes through: it will be easier to adapt GPU fitting to this new (also simpler) approach.
          • If it won't: I'll resume porting the road finding.