Rucio Meeting

Thursday 18 Jul 2024, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• August meeting schedule
  • Still fortnightly summer schedule (Until September)
  • Jul-25
  • Aug-01 Rucio 35 Release retrospective
  • Aug-08
  • Aug-15 Rucio 36 release roadmap
  • Aug-22
  • Aug-29

15:05 → 15:25 Community News & DevOps roundtable

• ATLAS
  • Minos stuck due to a regression in a PR
    • Fabio preparing a PR for this, will go in 35.1.0
• CMS
  • Status of new WebUI?
    • It's on integration for ATLAS, ssl issues resolved
    • Not fully stable yet
    • helm-chart & container available
      • Beware: ca-bundle needed for the node.js server (is documented in the container/chart)
  • Lots of unapproved rules
    • How to quickly process them?
      • No good way at the moment to approve/deny them in bulk
    • Advise users to not do single-file rules, but bulk them in datasets to optimise approval
• Fermilab
• DUNE
  • PR for VO tests
    • Still needing review
  • Big cleanup campaign of old files
    • ~80k files
      • COPYING replicas although all rules deleted
      • Can't be tombstoned
      • Under investigation
  • Consistency checking
    • Either via native auditor checking (ATLAS way) or with the consistency-checking toolbox (CMS way)
      • Did not converge both (yet)
    • Decide if sites provide dumps, or remote-scanning of namespace
• DaFab consortium
  • Setting up a DaFab Rucio instance @ CERN
• ESCAPE
• CERN IT
• ALTO
  • Preparing demo IETF hackathon
    • Can share result in next meeting
•  

15:25 → 15:55 Developers roundtable

client_refactor_6639_options.txt

Progress-2024-07-18.png

• Rucio 35 "Donkey and the Data Factory" roadmap
  • Todo
    • factorize duplicate messaging code into a common module or class #6423 [Alex]
  • In Progress
    • Review operations where SSL verification is disabled #6632 [Dimitrios]
    • Make new token support Multi-VO compatible #6406 [Dimitrios]
    • Evaluate all TODO and FIXME comments #6642 [Martin]
  • In Review
    
    • Finalize SQLAlchemy 1.0 -> 2.0 syntax migration #6640 [Erling]
      • Missing
        • core/replica.py
        • filter_engine.py

• • Done
    • SUBSCRIPTIONS_HISTORY integrity error #6292 [Cedric]
    • Use OpenID Connect Discovery #6414 [Dimitrios]
    • Rename internal API layer to gateway #6644 [Erling] 
    • overwrite_when_only_on_disk #6581 [Hugo]
    • Formatting / style guide #287 [Riccardo, Dimitrios, Maggie]
    • Collect functional RSE attributes into a single file #6624 [Dimitrios, Mira]
    • Remove all instances of mutable objects (e.g. empty lists or dictionaries) as default parameter values #5169 [Erling]
    • Make a table for all daemon and its functionality #296 [Hugo]
    • Webui 35 tracker 12/37 [Mayank]
    • Improving docstrings #6628 [Riccardo]
    • Type annotate the code #6588 [Riccardo]
    • JupyterLab version (v3 to v4) update #27 [Francesc, Enrique]
    • Create and populate common monitoring repository #6641 [Tim]
  • Delayed
    • Unable to Delete File DID via Undertaker #5154 [Alex]
    • Allow for token providers other than IAM #6630 [Dimitrios]
    • Make WebDAV protocols on par with GFAL #6633 [Dimitrios] 
    • Retire RSE limits #6629 [Erling, Dimitrios]
    • bridge the gap between running rucio in demo env and full production deployment #187 [Giovanni, Enrique, Hugo]
    • Prevent multiple Reaper threads from working on the same replicas #6512 [Hugo]
    • Prepare replacement of current policy import with policy packages #4798 [James]
      • ATLAS [Riccardo]
        • Package PR merged
        • Needs some work in the deployment
      • ESCAPE [Enrique]
        • Policy package now on cern gitlab
      • Belle II policy file not used, but is needed for the Belle II VO tests [Cedric]
        • Same for the ALAS VO test
    • Testing: Client Only tests #6599 [Mayank, Maggie]
    • Re-write and standardization of client command structure #6639 [Maggie]
      
      • Will require some additional work
    • Archive workflows for rules (Using an archive as a source) #6643 [Riccardo]
• Documentation corner
  • Versioning schema of secondary rucio products #NNNN [Enrique, Mayank]
  • Document activities used by rucio-daemons #326 [???]
  • Generate "Configuration parameters" documentation page automatically, to avoid mismatches between documentation and code #325 [???]
  • Error when trying to upload file with custom name containing slashes #336 [???]
  • Make a table for all daemon and its functionality #296 [Hugo]
  • Documentation (developer): Add guidelines for testing API-related changes #322 [???]
  • Documentation for RSE settings #293 [???]
  • Document how deletion occurs. #288 [Anil]
  • Introduce documentation on subscriptions #190 [Cedric]
  • bridge the gap between running rucio in demo env and full production deployment #187 [Hugo, Giovanni]
    • PR open! #349
    • Dimitris following the guide and provide feedback
  • Add instruction about DB partitioning #185 [Martin]
  • Add an FAQ-style entry aimed at users for STUCK rules #184 [Fabio]
  • Document environmental variables affecting the client #171 [Dimitrios]
  • Changing auth_type should invalidate current Rucio authn token #321 [???]
• Other topics
  • Client command structure
    • Discussion
    • Will expand the discussion and opinion collection to Mattermost (Town Square) and launch a poll
    • Decision in next Rucio meeting

15:55 → 16:00 AOB