Rucio Meeting

Thursday 21 Nov 2024, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• 36 Code Freeze tomorrow!

15:05 → 15:25 Community News & DevOps roundtable

• ATLAS
  • Kubernetes issues
    • Partly due to the IGTF CAs used in ATLAS
    • Nodes dropping at random points
    • Latest cluster template (1.31.1-1) include it, so we could get rid of the ATLAS implementation of it
  • Another issue happened due to this
    • Rate limit issue on dockerhub for some container images hosted on dockerhub
    • Possible to use cern proxy for it
    • But were hit for busybox containers used for logmessages
    • Possible to work around either in flux config or helm chart
    • Would be good to have a solution for everyone in helm-chart (global proxy config)
      • -> Issue #
      • To check: when to pull images and when to use it from cache
  • Token issue
    • Identified an issue with IAM / Rucio / FTS
      • Token-exchange enables client to break out of the token limitation of the orginal access token; Basically any token scope can be requested -> thus breaking out of the original scope limitation
      • Will be picked up by Token Task Force
• CMS
  • Collection replicas table issue
    • Forcing --deep, similar to ATLAS
      • This should be made the default in Rucio (36?)
    • Oracle cron job just trunacting col_rep_update table
  • Finisher memory growing a lot
    • Might be related to the growing col_rep_update table?
  • Two issues created, please check
• Fermilab DUNE, RUBIN, ...
  • DUNE
    • Switching SRM to DAVS
      • Test if rest api works
    • delete-file / erase file
      • #7150
        • No update yet
    •  Tokens
      • CiLogon preparing a patch to be able to pass tests
  • RUBIN
    • 35.6.0
      • SLAC security requested to update due to older (?) versions of httpd and openssl in the container
    • Connection issue (discussed on mattermost)
      • Disappeared with new version of openssl in 35.6
    • hermesK (including kafka) 
      • Failing sending messages
• DaFab
  • Preparation for HIPEAC'25 workshop
  • Metadata updates & challenge to keep it backwards compatbile
• ESCAPE
  • rucio-ui not accessible outside of CERN
• CERN IT
• RAL / MultiVO
• IHEP
  • HERD experiment on 1.29 LTS version
    • Update by end of the year!
• HEP-CCE
  • Ben setting up Rucio standalone with token for Rucio Auth
    • Globus tokens for globus auth
    • Switching to X509  -> Is enough for Globus transfer testing

15:25 → 15:55 Developers roundtable

Progress-2024-11-21.png

• 36 "Donkey Unchained" roadmap
  • Todo
  • In Progress
    • Make new token support Multi-VO compatible #6406 [Dimitrios]
    • Review operations where SSL verification is disabled #6632 [Dimitrios]
    • Retire RSE limits #6629 [Erling, Dimitrios]
    • Prevent multiple Reaper threads from working on the same replicas #6512 [Hugo]
  • In Review
    • factorize duplicate messaging code into a common module or class #6423 [Alex] 
    • Re-write and standardization of client command structure #6639 [Maggie]
      • Let's merge, see what comments we get (It's backwards compatible anyway)
      • replica subcommand might need some updates
      • Will need some review of the implementation, most feedback/focus was on the command structure/syntax
      • Is not an experimental feature, since it will become the default way/structure of interacting with Rucio!
      • Will perhaps only merge for FINAL (Since RCn will only be tested on server, not client)
    • Prepare replacement of current policy import with policy packages #4798 [James, Riccardo, Martin]
    • Refactor all ATLAS-related code into ATLAS policy package #7027 [Riccardo]
    • Better define unit tests and integ tests #7007 [Riccardo]
      • Few PRs pending review for this
    • Improve test coverage #7008 [Riccardo]
      • Few PRs pending review for this
  • Done
    • Allow permissions to optionally return a message #6580 [James]
    • WebUI Tracker 36 [Mayank, Viktoriaa]
  • Delayed
    • Make WebDAV protocols on par with GFAL #6633 [Dimitrios]
    • Implement new token authentication for download #7029 [Dimitrios]
      • Does not need to wait for a major release, but will not be in 36.0 (Aiming for 36.1/2)
    • Allow for token providers other than IAM #6630 [Dimitrios]
      • With recent developments, possibly no Token Provider plugins are needed, pause this for the moment
      • Wait for example implementation with CiLogon and for the ongoing offline_access discussion with IAM
    • Create a new API endpoint for the clients to request tokens #6638 [Dimitrios]
      • Will come in 36.1/2
    • Improve docstrings #6628 [Riccardo]
    • Evaluate archive workflows for rules (Using an archive as a source) #6643[Riccardo]
• Documentation corner DOCUSPRINT
  • Document activities used by rucio-daemons #326 [???]
  • Generate "Configuration parameters" documentation page automatically, to avoid mismatches between documentation and code #325 [???]
  • Error when trying to upload file with custom name containing slashes #336 [???]
  • Make a table for all daemon and its functionality #296 [Hugo]
  • Documentation (developer): Add guidelines for testing API-related changes #322 [???]
  • Documentation for RSE settings #293 [???]
  • Document how deletion occurs #288 [Anil]
    • PR #295 merged
  • Introduce documentation on subscriptions #190 [Cedric]
  • Add instruction about DB partitioning #185 [Martin]
  • Add an FAQ-style entry aimed at users for STUCK rules #184 [Fabio]
  • Changing auth_type should invalidate current Rucio authn token #321 [???]
• Other topics
  • •  

15:55 → 16:00 AOB