Rucio Meeting

Thursday 2 Sept 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• Rucio website Team introduction
  • For now I added/asked all component leads
  • All other developers are also very welcome to add their names/intro!
    • Please submit a PR to https://github.com/rucio/rucio.github.io
• September release schedule
  • 1.26.4 Sep 6
  • 1.23.16 LTS Sep 6
  • 1.26.5 Sep 27
• Metadata tests in Belle II presentation moved to Sep 16
• CHEP reminder
• Next week no meeting (Jeûne genevois)
• Sep 16
  • GSOC presentation: New protocols for exascale data management with Rucio

15:05 → 15:15 Community News & DevOps roundtable

• ATLAS
  • Reaper certificates
    • For now including a bunch of certificates via a k8s secrets
      • This is limited to 1MB, we are now hitting that limit
      • One solution would be OPA gatekeeper
        • Reaper would mount certs via OPA gatekeeper
        • ESCAPE using this already
  • Token deadlock issue due to a query running with two different query hints
    • Fixed by running the same query
• CMS
• Fermilab
  • Running tests on 1.26
  • Upgrading Icarus to 1.26 soon
    • DUNE next
• Belle II
  • Popularity infrastructure running
    • Tracer importing traces
    • DIRAC datamanager used to fetch data from grid
      • Thinking about moving to downloadclient
      • Needs fix in client to allow B2 naming convention (PR there)
• LDMX
  • Would be good to be able to run a server with ALL endpoints
    • rucio.cfg switch to be able to import all
    • Martin will add note to the release notes
  • Running alembic
    • Alembic did not recognize the revision the DB is in, tried to apply all
    • Running without alembic (--sql) works
• ESCAPE
  • Not flux+helm but upstream flux
  • 1.26 upgrade
    • Webui still on 1.25 due to https://github.com/rucio/rucio/issues/4810
    • OIDC token issue (Template file missing?)
      • REST and WebUI OIDC works
      • Possibly just the template file missing in the packaging
  • Datalake as a Service
    • Now complete
    • Finished Rucio upload functionality
    • Datalake upload functionality also works (TPC based)
• SKAO

15:15 → 15:20 Hot topics

15:20 → 15:35 GSOC: Rucio and CS3API to enable data management for the ScienceMesh cloud

Speaker: Rahul Chauhan

Rucio and CS3APIs_GSoC.pdf

15:35 → 15:50 Metadata tests in Belle II [Moved to September 16]

Speaker: Cedric Serfon (Brookhaven National Laboratory (US))

15:50 → 15:55 Developers roundtable

Rucio 1.27 "Batdonkey v. Superdonkey" priority followup

• In Progress
  • Auditor overhaul #3437 [Dimitrios]
    • Will discuss with Dimitrios once he is back
  • rucio.cfg vs config table #2630 [Mario, David]
    • Needs review
  • Logging review #4220 [Martin, All comp leads]
    • No news
  • Disentangle fts3 specific code from conveyor and move to transfertool #857 [Radu]
    • Work ongoing
    • One open PR for submitter-stager merge
    • One other PR already prepared, will submit once previous merged
  • helm-charts release management #4794 [Radu, Eric, Martin]
    • Eric submitted comments to the issue
    • Switch to the new naming scheme first, thus next release would be 1.26.0
    • Needs to be described on the documentation
    • Github action needs to be upgraded to look into separate release branches
      • Probably not a big change
  • Quality of Service #3419 [Matt]
    • Demonstrator for sotrage-issued QoS changes for BNL MAS
    • Matt should have dev changes done in the coming weeks
• Todo
  • Get SSO Login working [Rizart]
  • Down-scoped tokens for user interactions #4791 []
  • Optimize database interactions #4793 [Martin, Mario, Radu]
  • Prepare replacement of current policy import with policy packages #4798 [James, Martin]
  • Rename Daemons to commonly understandable names #4795 [Martin]
  • Versioning for REST API #4796 [Ben, Martin]
  • Enabling tests for different policy package #3878 [Mayank]
• Done
• Delayed

Developer roundtable

• Docker auto builds [Mayank]
  • Will not use docker auto builds anymore, instead have GH actions build and push the containers
  • Workflow PR is in place
  • Secrets for Docker are in place
  • Right now builds daemons, clients, server; will add the other auto-builds as well
• Enabling http on xrood containers
• Changing CI oracle-express database from 11 to 18
  • PR done, merge asap

15:55 → 16:00 AOB