Rucio Meeting

Thursday 31 Aug 2023, 15:00 → 16:00 Europe/Zurich

Dimitrios Christidis (CERN)

1 News

• Meeting schedule:
  • No meeting on 7th September (Jeûne Genevois).
  • Next meeting on 14th September.
  • Resuming weekly meetings (as opposed to bi-weekly during the summer).
• September release schedule
  • 1.29.14 LTS Sep 14
  • 32.? Sep 14
  • 32.? Sep 25
• WebUI presentation by Mayank and Anton in meeting on Sep 14

2 Community News & DevOps roundtable

• ATLAS
  • Have applied a patch which forces deep=True for all list-dataset-replicas requests.
    • This might remove the necessity for the collection_replicas table.
    • Initial results show poor performance on the database; however, it could be due to other activity.
    • Will continue to monitor and report in the next meeting.
    • This effort is related to the work on the Abacus daemon.
• CMS
  • Follow-up on Kubernetes nodes becoming unstable:
    • Overflow of logs (related to CERN and MONIT, not Rucio).
  • Investigated the requirements for updating to Rucio release 32.
• BelleII
  • Poor deletion performance when using too many threads:
    • Improved by applying some change on the PostgreSQL partitioning.
• Fermilab/ICARUS/Rubin
  • First draft of policy package.
  • Upgrade from release 1.29 to 32.
  • Custom version of Hermes daemon with Kafka support.
  • Issue with Automatix daemon:
    • Unexpected authentication errors (reported on Mattermost).
• DUNE
  • Work for WLCG Data Challenge.
  • Had to switch from regular to greedy deletion:
    • Appearance of dark data. Interested in the CMS-style consistency enforcement.
• ESCAPE
  • Everything appears to be working fine on the new Rucio instance.
• ALTO
  • Would like to discuss transfer visualisation in the next meeting.
  • Should also get in touch with Radu.
• KM3NeT
  • Gone through the procedure for setting up Rucio (running on Kubernetes).
    • Some room for improvement in the documentation.

3 Developers roundtable

Progress-2023-08-31.png

• Rucio 33 "Eternal Sunshine of the Donkey's Mind" roadmap
  • In Progress
    • Design and implement a token cache module #6183 [Dimitrios]
    • Introduce the RSE decommissioner daemon #6321 [Yutaro, Dimitrios]
    • WebUI Release 33 page tracker #301 [Mayank/Anton]
  • In Review
    • Unable to Delete File DID via Undertaker #5154 [Anton]
    • Incorporate IAM-based testing for token workflows #6188 [Eraldo, Dimitrios]
  • Todo
    • Design and implement a token request module #6187 [Dimitrios]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • foreign key error on deleting dids in reaper #5733 [Martin]
    • Review state of the Light Reaper #6317
    • Review lost data handling
    • Tests cleanup #6212
    • Transfer status visual representation #6189 [Radu]
    • Dynamic distance between RSEs #5012 [Radu]
    • revisit submitter source selection strategy #5776 [Radu]
  • Done
• Documentation corner
  • Document environmental variables affecting the client #171 [Dimitrios]
  • Documentation and dev guidelines for Mypy type annotations #116 [Mayank, Martin]
  • Create developers testing guide in the documentation #177 [Mayank]
  • Create a server/daemon installation howto #178 [Mayank]
    • Ubuntu-specific PR #215
  • Improve documentation on rucio.cfg vs configuration table #183 [Radu]
  • Add an FAQ-style entry aimed at users for STUCK rules #184 [Fabio]
  • Add instruction about DB partitioning #185 [Martin]
  • bridge the gap between running rucio in demo env and full production deployment #187 [Radu]
  • Introduce documentation on subscriptions #190 [Cedric]
  • Page about the different event types and messages (ActiveMQ) #226 [Eraldo]
    • Waiting on comments from Martin

4 AOB