Rucio Meeting

Thursday 14 Nov 2024, 15:00 → 16:00 Europe/Zurich

Dimitrios Christidis (CERN)

1 News

• Release schedule
  • 35.? on 18 November
  • 36 code freeze on 22 November
• DocuSprint 2024
  • Please register

2 Community News & DevOps roundtable

• ATLAS
  • Token scope offline_access 
    • IAM is creating and storing refresh tokens, even though they are not returned to Rucio (GH issue)
    • We were asked to removed the offline_access scope from the token request.
    • But firtst, we must verify with FTS that their workflows will still continue to work, and devise an alternative method to communicate whether to assume the lifecycle of the token or not.
• CMS
  • Issue with recreating rules:
    • Context: a rule that has already expired exists, and before the rule has been processed by the Cleaner, another request to create the rule happens
    • This is mostly in the context of a WFMS, though it might apply to individual users as well.
    • Can the expired_at be part of the unique constraint? To discuss in a GitHub issue.
  • Question about source selection strategies. To ask on Mattermost.
• Fermilab
  • Adapting the policy packages to the latest upstream
• RUBIN
  • Judge Injector takes too much time (25-35 minutes for ??K files)
    • The internal metrics suggest the delay may be in the apply_rule() function
  • Rucio permissions
    • Many are for just account=root and admin=True. Can they be simplified?
    • But also: to get a better overview fo what administrators versus regular users are allowed to do. To consider expanding the documentation.
• DaFab
  • Demonstration of the new metadata mechanism at a workshop this winter
• ESCAPE
  • Addressing issues with OIDC tokens
    • Fixed the token-exchange workflow with IAM.
  • Setting up the Rucio 34 Web UI
  • Preparing the deployment of Rucio 35
• CERN IT
  • Setting up a pilot instance for the AMS experiment, late this year or early next year
• PIC
  • Rucio instance for LST
    • Update from Rucio 32 to 35, plus additional configuration

3 Developers roundtable

• 36 "Donkey Unchained" roadmap
  • Todo
    • Implement new token authentication for download #7029 [Dimitrios]
      • Depend on #6638, but is also top priority
  • In Progress
    • factorize duplicate messaging code into a common module or class #6423 [Alex]
    • Allow for token providers other than IAM #6630 [Dimitrios]
      • Depend on #6406
    • Make new token support Multi-VO compatible #6406 [Dimitrios]
    • Create a new API endpoint for the clients to request tokens #6638 [Dimitrios]
      • Will be experimental, but is top-priority
    • Review operations where SSL verification is disabled #6632 [Dimitrios]
      • Almost done, just some doc missing
    • Retire RSE limits #6629 [Erling, Dimitrios]
    • Prepare replacement of current policy import with policy packages #4798 [James, Riccardo, Martin]
      • Belle II policy almost DONE
        • Created a temporary policy package for belle 2 for VO tests: 

          https://github.com/rucio/temporary-belle2-policy-package

      • ATLAS policy DONE
    • WebUI Tracker 36 [Mayank, Viktoriaa]
      • All PR related to X.509 authentication are now merged
      • Pages are still being worked on
    • Refactor all ATLAS-related code into ATLAS policy package #7027 [Riccardo]
    • Better define unit tests and integ tests #7007 [Riccardo]
      • Few PRs pending review for this
    • Improve test coverage #7008 [Riccardo]
      • Few PRs pending review for this
    • Prevent multiple Reaper threads from working on the same replicas #6512 [Hugo]
  • In Review
    • Re-write and standardization of client command structure #6639 [Maggie]
      • Let's merge, see what comments we get (It's backwards compatible anyway)
      • replica subcommand might need some updates
      • Will need some review of the implementation, most feedback/focus was on the command structure/syntax
      • Is not an experimental feature, since it will become the default way/structure of interacting with Rucio!
  • Done
    • Allow permissions to optionally return a message #6580 [James]
  • Delayed
    • Make WebDAV protocols on par with GFAL #6633 [Dimitrios]
    • Improve docstrings #6628 [Riccardo]
    • Evaluate archive workflows for rules (Using an archive as a source) #6643[Riccardo]
• Documentation corner
  • Document activities used by rucio-daemons #326 [???]
  • Generate "Configuration parameters" documentation page automatically, to avoid mismatches between documentation and code #325 [???]
  • Error when trying to upload file with custom name containing slashes #336 [???]
  • Make a table for all daemon and its functionality #296 [Hugo]
  • Documentation (developer): Add guidelines for testing API-related changes #322 [???]
  • Documentation for RSE settings #293 [???]
  • Document how deletion occurs #288 [Anil]
    • PR #295 merged
  • Introduce documentation on subscriptions #190 [Cedric]
  • Add instruction about DB partitioning #185 [Martin]
  • Add an FAQ-style entry aimed at users for STUCK rules #184 [Fabio]
  • Changing auth_type should invalidate current Rucio authn token #321 [???]
• Other topics
  • CILogon implementation #7161
    • DUNE will comment on their progress
  • Data injection tool (for Data-Challenge-style load testing)
    • Presentation by Xuantong
    • Currently collecting the requirements

4 AOB