We deployed Indico v3.2. See our blog post for details on the changes.

Rucio Meeting

4/S-030 (CERN)



Show room on map
Martin Barisits (CERN)
Rucio Development Meeting
Zoom Meeting ID
Martin Barisits
Alternative host
Mario Lassnig
Useful links
Join via phone
Zoom URL
    • 3:00 PM 3:05 PM
      News 5m
      • June meeting schedule
        • June 02
        • June 09
        • June 16 CANCELLED
        • June 23
      • Next week no meeting! (Holiday @ CERN)
      • Ticket review
        • eMail to component leads out for 1:1 meetings
    • 3:05 PM 3:20 PM
      Community News & DevOps roundtable 15m
      • ATLAS
        • Got Amazon S3 commercial cloud to run within WLCG infrastructure
          • Put in Rucio documentation together with the S3 article?
          • Very specific documentation to Amazon
      • CMS
        • Started upgrade process to 1.28
          • So far on INT
        • Issue with DNs which have ',' inside them
          • Fixed by using LegacyDN
            • Legacy won't support unicode, special characters needs to be escaped, etc.
          • Added to helm-charts and containers (Option)
        • Getting Globus integrated
      • Fermilab/DUNE/Ikarus/Rubin
        • DUNE waiting on DataChallenge
        • Ikarus accessing service off-site
        • Everyone
          • Looking at argo CD (Instead of Flux)
            • Argo CD specifically useful for multi-cluster setups
        • Dennis Lee joining Rucio efforts at FNAL
      • DUNE
        • Data challenge
          • Staying with 1.26 LTS
          • Monitoring transactions and trans-atlantic throughput
        • QoS
          • Hope was to make more progress in DOMA QoS workgroup
          • Change of Pledging system is a barrier for established experiments
          • DUNE will critically rely on QoS
            • Need to find a way forward to make it a usable product for everyone
        • Token timeline in Rucio
          • DUNE timeline driven by RHEL goes end of service (2024)
        • Load into Rucio a custom proximity map
          • Especially for sites which do not have storage and thus no distance entry in Rucio
      • Belle II
        • Waiting on migration for 1.28
          • Migration to 1.29 pending then on python 3 migration of DIRAC (End of summer)
        • Discussion on archival features for metadata
          • Discussion between Rob, Cedric, Martin yesterday, Cedric will work on it for 1.29
      • ESCAPE
        • Moved from 1.28.2 to 1.28.4
        • Looks like after submitting a rule the submission of the transfer takes longer
          • Took 5h to submit a transfer
            • Although daemons looked not very busy
        • PR to fix Jupyterlab extension
          • Error was with the change of DIDTypes from strings to enums
      • SKAO
      • MultiVO/RAL
        • Issue #5319 for implementing posix protocol with symlink functionality
          • Unittest would be beneficial
    • 3:20 PM 3:25 PM
      Hot topics 5m
    • 3:25 PM 3:55 PM
      Developers roundtable 30m
      • Rucio 1.29 "Into the donkeyverse" progress tracking
        • In Progress​​
          • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
          • Fix client API documentation page #5444 [Joel]
            • Basically done, some edge cases
          • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
          • Ticket Cleanup Marathon [Martin, All]
          • Reduce logging of server & daemons #5449 [Mayank]
            • Now access to ES
            • Started to populate dashboards about frequency distribution of log patterns
          • Authentication workflows in WebUI 2.0 #5135 [Mayank]
            • stuck on setting up base repo for webui
          • Add python 3.10 tests to CI framework. #5145 [Mayank]
            • Fedora images are ready for 3.10
            • Stopped to work on CC7 - not worth it
            • Currently looking into centos stream 8
          • Setup WebUI repository for external collaboration #w/12 [Mayank]
            • Update next week
          • Introduce heartbeat handler to all daemons #5443 [Radu, ALL]
            • Started to move some daemons to the new daemon_handler
          • Further integration of temporary tables to other major workflows #5441 [Radu]
          • Further OIDC and SSO optimization and documentation #5437 [Rizart]
            • Ticket done, now looking on related ticket which will be submitted as one
          • Migration of SQLAlchemy queries to new query syntax (1.4 -> 2.0) #5247 [Yuyi, ALL]
        • In Review
          • Hermes 2 evolution #4610 [Cedric]
            • Comments to be included
          • Calculation of container length and events #5422 [Radu]
            • Difficult to calculate proper statistics for containers without performance degredation
            • Open PR, but unclear if it fully closes the issue
        • Done
          • re-evaluate the greedy = 1TB constant #5358 [Cedric]
          • Clean dead code in the probes (ATLAS and common) #83 [Eric, Dimitrios]
          • boto vs boto3 #5346 [James, Mario]
          • mypy static code analysis #4455 [Mayank, Ben, Joel]
            • Discussion for type checking and documentation for developers guide coming
          • Flag transfer errors coming from tape sources #5410 [Radu]
          • Re-factor/cleanup deletion workflow #5442 [Radu]
          • Add session object to API layer #5439 [Radu]
          • Remove author list from headers [Martin, Joel]
        • Todo
          • Create a contributor guide for probes #84 [Dimitrios]
          • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
          • Create a server/daemon installation howto #5445 [Joel]
          • Add symlinks for the new daemon names #5131 [Joel]
            • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
          • Reduce core component tickets to less than 10 [Mario, Martin, Gabriele]
          • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
          • Unable to Delete File DID via Undertaker #5154 [Martin]
          • Removal of Python 2.7 dependencies from the code #5435 [Martin]
          • Reduce rule tickets to les than 10 [Martin]
          • Create developers testing guide in the documentation #5452 [Mayank, Joel]
            • Ben working on sub-task to test locally
          • Remove direct DB dependence from the WebUI #4319 [Mayank]
          • Analyze feedback on UI Survet #w/13 [Mayank]
          • Remove full_mode switch from conveyor-receiver #5453 [Radu]
          • Exchange of function-based indices with normal indices #5440 [Radu, Mario, Martin]
          • Re-work/cleanup distance table #5454 [Radu]
          • Create documentation page for S3-type storage #5450 [Rizart]
            • Exists as GGDoc (#5284)
          • Add support for S3 virtual hosted buckets #5451 [Rizart]
            • Test on-premise
            • Ongoing support issue with AWS/Fresno X.509 shenanigans
          • Update/Re-design core.meta module #5224 [Rob]
          • Merge list_dids and list_dids_extended methods #5448 [Rob]
      • Container linux distribution
        • CC7 images harder to manage (Python 3.10++)
        • CS8 and CS9 already in testing
        • Any strong preference for an OS for the official rucio images?
          • Can it be something else than CentOS?
          • ATLAS -> CentOS (Stream)
          • Fermilab -> CentOS
          • CMS offline jobs moving to alma8
            • Otherwise no preference for the actual rucio container
        • Python versions - 3.10 needed?
          • Not strictly needed, but would be good eventually
    • 3:55 PM 4:00 PM
      AOB 5m
      • Rucio workshop 2022
        • For now operating assumption, due to Covid and travel restrictions we will have the workshop at CERN in September/October
        • However, if any institute would like to host the workshop please get in contact with me asap