We deployed Indico v3.2. See our blog post for details on the changes.

Rucio Meeting

Europe/Zurich
4/S-030 (CERN)

4/S-030

CERN

30
Show room on map
Martin Barisits (CERN)
Videoconference
Rucio Development Meeting
Zoom Meeting ID
413496641
Host
Martin Barisits
Alternative host
Mario Lassnig
Passcode
28849311
Useful links
Join via phone
Zoom URL
    • 3:00 PM 3:05 PM
      News 5m
      • June meeting schedule
        • June 02
        • June 09
        • June 16 CANCELLED
        • June 23
      • Next week no meeting! (Holiday @ CERN)
      • Ticket review
        • eMail to component leads out for 1:1 meetings
    • 3:05 PM 3:20 PM
      Community News & DevOps roundtable 15m
      • ATLAS
        • Got Amazon S3 commercial cloud to run within WLCG infrastructure
          • Put in Rucio documentation together with the S3 article?
          • Very specific documentation to Amazon
      • CMS
        • Started upgrade process to 1.28
          • So far on INT
        • Issue with DNs which have ',' inside them
          • Fixed by using LegacyDN
            • Legacy won't support unicode, special characters needs to be escaped, etc.
          • Added to helm-charts and containers (Option)
        • Getting Globus integrated
      • Fermilab/DUNE/Ikarus/Rubin
        • DUNE waiting on DataChallenge
        • Ikarus accessing service off-site
        • Everyone
          • Looking at argo CD (Instead of Flux)
            • Argo CD specifically useful for multi-cluster setups
        • Dennis Lee joining Rucio efforts at FNAL
      • DUNE
        • Data challenge
          • Staying with 1.26 LTS
          • Monitoring transactions and trans-atlantic throughput
        • QoS
          • Hope was to make more progress in DOMA QoS workgroup
          • Change of Pledging system is a barrier for established experiments
          • DUNE will critically rely on QoS
            • Need to find a way forward to make it a usable product for everyone
        • Token timeline in Rucio
          • DUNE timeline driven by RHEL goes end of service (2024)
        • Load into Rucio a custom proximity map
          • Especially for sites which do not have storage and thus no distance entry in Rucio
      • Belle II
        • Waiting on migration for 1.28
          • Migration to 1.29 pending then on python 3 migration of DIRAC (End of summer)
        • Discussion on archival features for metadata
          • Discussion between Rob, Cedric, Martin yesterday, Cedric will work on it for 1.29
      • ESCAPE
        • Moved from 1.28.2 to 1.28.4
        • Looks like after submitting a rule the submission of the transfer takes longer
          • Took 5h to submit a transfer
            • Although daemons looked not very busy
        • PR to fix Jupyterlab extension
          • Error was with the change of DIDTypes from strings to enums
      • SKAO
      • MultiVO/RAL
        • Issue #5319 for implementing posix protocol with symlink functionality
          • Unittest would be beneficial
    • 3:20 PM 3:25 PM
      Hot topics 5m
    • 3:25 PM 3:55 PM
      Developers roundtable 30m
      • Rucio 1.29 "Into the donkeyverse" progress tracking
        • In Progress​​
          • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
          • Fix client API documentation page #5444 [Joel]
            • Basically done, some edge cases
          • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
          • Ticket Cleanup Marathon [Martin, All]
          • Reduce logging of server & daemons #5449 [Mayank]
            • Now access to ES
            • Started to populate dashboards about frequency distribution of log patterns
          • Authentication workflows in WebUI 2.0 #5135 [Mayank]
            • stuck on setting up base repo for webui
          • Add python 3.10 tests to CI framework. #5145 [Mayank]
            • Fedora images are ready for 3.10
            • Stopped to work on CC7 - not worth it
            • Currently looking into centos stream 8
          • Setup WebUI repository for external collaboration #w/12 [Mayank]
            • Update next week
          • Introduce heartbeat handler to all daemons #5443 [Radu, ALL]
            • Started to move some daemons to the new daemon_handler
          • Further integration of temporary tables to other major workflows #5441 [Radu]
          • Further OIDC and SSO optimization and documentation #5437 [Rizart]
            • Ticket done, now looking on related ticket which will be submitted as one
          • Migration of SQLAlchemy queries to new query syntax (1.4 -> 2.0) #5247 [Yuyi, ALL]
        • In Review
          • Hermes 2 evolution #4610 [Cedric]
            • Comments to be included
          • Calculation of container length and events #5422 [Radu]
            • Difficult to calculate proper statistics for containers without performance degredation
            • Open PR, but unclear if it fully closes the issue
        • Done
          • re-evaluate the greedy = 1TB constant #5358 [Cedric]
          • Clean dead code in the probes (ATLAS and common) #83 [Eric, Dimitrios]
          • boto vs boto3 #5346 [James, Mario]
          • mypy static code analysis #4455 [Mayank, Ben, Joel]
            • Discussion for type checking and documentation for developers guide coming
          • Flag transfer errors coming from tape sources #5410 [Radu]
          • Re-factor/cleanup deletion workflow #5442 [Radu]
          • Add session object to API layer #5439 [Radu]
          • Remove author list from headers [Martin, Joel]
        • Todo
          • Create a contributor guide for probes #84 [Dimitrios]
          • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
          • Create a server/daemon installation howto #5445 [Joel]
          • Add symlinks for the new daemon names #5131 [Joel]
            • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
          • Reduce core component tickets to less than 10 [Mario, Martin, Gabriele]
          • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
          • Unable to Delete File DID via Undertaker #5154 [Martin]
          • Removal of Python 2.7 dependencies from the code #5435 [Martin]
          • Reduce rule tickets to les than 10 [Martin]
          • Create developers testing guide in the documentation #5452 [Mayank, Joel]
            • Ben working on sub-task to test locally
          • Remove direct DB dependence from the WebUI #4319 [Mayank]
          • Analyze feedback on UI Survet #w/13 [Mayank]
          • Remove full_mode switch from conveyor-receiver #5453 [Radu]
          • Exchange of function-based indices with normal indices #5440 [Radu, Mario, Martin]
          • Re-work/cleanup distance table #5454 [Radu]
          • Create documentation page for S3-type storage #5450 [Rizart]
            • Exists as GGDoc (#5284)
          • Add support for S3 virtual hosted buckets #5451 [Rizart]
            • Test on-premise
            • Ongoing support issue with AWS/Fresno X.509 shenanigans
          • Update/Re-design core.meta module #5224 [Rob]
          • Merge list_dids and list_dids_extended methods #5448 [Rob]
      • Container linux distribution
        • CC7 images harder to manage (Python 3.10++)
        • CS8 and CS9 already in testing
        • Any strong preference for an OS for the official rucio images?
          • Can it be something else than CentOS?
          • ATLAS -> CentOS (Stream)
          • Fermilab -> CentOS
          • CMS offline jobs moving to alma8
            • Otherwise no preference for the actual rucio container
        • Python versions - 3.10 needed?
          • Not strictly needed, but would be good eventually
    • 3:55 PM 4:00 PM
      AOB 5m
      • Rucio workshop 2022
        • For now operating assumption, due to Covid and travel restrictions we will have the workshop at CERN in September/October
        • However, if any institute would like to host the workshop please get in contact with me asap