Rucio Meeting

Thursday 1 Sept 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• Meeting schedule
  • Back to weekly meetings!
  • Next week no meeting! (Jeune Genevois)
• September release schedule
  • 1.29.4 Sep 5
  • 1.29.5 Sep 19
• Rucio 1.30 Release checkpoint 1
  • September 15!

15:05 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Regression in the recent self-managed QoS feature
    • By accident all transfers were submitted including a STAGING request
    • Hotfix was submitted quite quickly disabling this feature
  • Issue with subscriptions
    • Caused by malformed historic subscriptions in the database
  • list_replicas with specific RSE expression + temporary tables activated
    • rse_expression needs to resolve to >4 RSEs
    • Rucio returns wrong, inconsistent, result
    • Fixed in next release
  • #5842 legacy vs RFC-conform format
    • IAM gives out DNs in RFC format, while VOMS does legacy format
    • There will be a probe in 1.30 taking care of this
      • RFC -> Legacy conversion will be removed
• CMS
  • 1.29.3 currently
  • bad_replica declaration very important
    • Ongoing discussion on Slack
• Belle II
  • 1.28.7
  • For 1.29 still waiting for Py3 environment/compatibility
    • --> Beginning of October
• Fermilab/DUNE/ICARUS/...
  • Need to upgrade to 1.29
    • Passing scope to surl algorithm is fundamental to DUNE
    • Be aware that 1.26 transfers might not be compatible directly in 1.29
      • Might need to drain transfers in a 1.28 daemon
• DUNE
  • metacat needs both name and scope, so it does require the scope in the SURL algorithm
  • Next week, second phase of data challenge
• ESCAPE
  • Summer student working on integration of Rucio and reana
    • Progressing well
  • Authentication errors on Rucio instance
    • Need to check

15:20 → 15:35 Rucio Testing Framework (Introduction and Updates)

Speakers: Mayank Sharma (University of Texas at Arlington (US)), Benedikt Ziemons (FH Aachen University of Applied Sciences (DE))

Rucio DonkeyRider Introduction

15:35 → 15:55 Developers roundtable

Progress-2022-09-01.png

• Rucio 1.30 "The Donkeynator" progress tracking
  • In Progress​​
    • Introduce heartbeat handler to all daemons #5443 [Cedric, Dimitrios, Mario]
      • Missing daemons: hermes, atropos, replicarecoverer
        • replicarecoverer Christoph will do it, Cedric will double check
          • Christoph will be back to work on this in 2-3 weeks
        • atropos Dimitrios will check asap
    • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
    • Exchange of function-based indices with normal indices #5440 [Mario, Martin, Radu]
      • Tombstone will be the first
    • Unable to Delete File DID via Undertaker #5154 [Martin]
      • More complicated than initially thought; still under investigation
    • Create a server/daemon installation howto #5445 [Joel]
    • rework conveyor-throttler #5805 [Radu]
    • Switch to pytest [Radu?]
      • Needs discussion
    • Release webUI 2.0 beta version [Mayank]
  • In Review
  • Done
    • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
    • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
  • Todo
    • Refactor the different test workflows to a common one [Ben, Mayank]
    • Create a contributor guide for probes #84 [Dimitrios]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Create developers testing guide in the documentation #5452 [Joel, Mayank]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Reduce rule tickets to les than 20 [Martin]
    • foreign key error on deleting dids in reaper #5733 [Martin]
    • Rules on containers in state OK but not all the files from the containers have locks #5447 [Martin]
    • Reduce logging of server & daemons #5449 [Mayank]
    • Add python 3.10 tests to CI framework. #5145 [Mayank]
    • rework prometheus/statsd metric names #5804 [Radu]
    • Re-work/cleanup distance table #5454 [Radu]
    • Create documentation page for S3-type storage #5450 [Rizart]
      • Exists as GGDoc (#5284)
    • Add support for S3 virtual hosted buckets #5451 [Rizart]
      • Test on-premise
      • Ongoing support issue with AWS/Fresno X.509 shenanigans
    • Merge list_dids and list_dids_extended methods #5448 [Rob]
  • Delayed
• Other topics
  •  

15:55 → 16:00 AOB