Rucio Development Meeting

Thursday 4 Feb 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• New team member: Radu
• GSOC Projects
  • rclone
  • CS3
• CERN Summer student submission
• Release schedule
  • 1.24.3 tomorrow
  • 1.24.4 Feb-15
  • 1.24.5 Feb-22
  • Code Freeze from Feb-24
    • 1.25.0rc1 Feb-25
    • 1.25.0 Mar-1 ... Mar-5
• Rucio Community CHEP Paper
  • Submitted to ATLAS CSC
    • Now under review
  • Submission to CHEP: Feb-28

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Trials with volatile storage in Edinburgh
    • Operational impact of out of warranty disks
  • Exclusion thresholds for reaper2 set to 10
    • Better deletion performance
• CMS
  • 1.24 works now after some fixes
  • Dark reaper now running on K8s
• Belle II
  • Things working fine
  • Still on 1.23
• DUNE/Edinburgh
  • Lightweight clients -> PR later this week
  • Dedicated session for future Rucio-Dev meeting?
  • rclone GSOC project might be interesting
• ESCAPE
  • SQLAlchemy name length warning for Oracle 19 #4294
  • Table growing of updated_col_rep table
    • New Abacus PR should help with that a bit

15:20 → 15:30 Hot topics

• SQLAlchemy enum issues
  • Several issue due to recent changes in SQLalchemy from constants to enums
    • Needed changes in the entire codebase, some changes were hard to spot and forgotten
  • Lessons learned:
    • More testcases - We should remind ourselves when developing how important proper testing is.
    • After the first issues are detected: Deeper code-dive needed to find others
  • Things which could help
    • Python type method annotations
    • No need to assert everything, but type annotations would already help

15:30 → 15:55 Developers roundtable

• Rucio 1.25 "Rat-Donkey" release followup
  • In Progress
    • Stronger integration of Globus Online transfertool #4216 [Matt, Ben]
      • PRs done, finalizing tomorrow
    • rucio.cfg vs config table #2630 [Mario]
      • Circular dependency between core-config and common-config
        • Not yet resolved
        • Will need a new module to resolve
    • Remove webpy endpoints and dependency #4044 [Ben]
      • Start using FLASK on integration server of ATLAS next week
        • Waiting for 1.24.3 release to test
      • Completed OIDC flask template
      • Removal of web.py with 1.25 should still be fine
    • Quality of Service #3419 [Mario, Martin, Rizart, *]
    • Deprecate reaper1 #4213 [Martin]
      • Should be done by next week, still some issues with testcases
      • Reaper2 testcase exposed a few differences to reaper1
        • MaxFilesBeingDeleted setting does not work anymore -- remove
        • Deleting until exactly the threshold does not work in reaper2
    • Logging review #4220 [Component Leads]
      • Internal deadline: Feb-11
      • Added decorated logger added to all daemons/server
        • Now prefix string handed through to core layer as well
    • Identify and cleanup unused functionality and code #4221 [Component Leads]
      • Internal deadline: Feb-11
      • Most likely not happening with 1.25
      • pytest coverage - need to try
    • Versioned (History) Tables should be defined explicitly #2063 [Martin]
  • Todo
    • Test new rule mode and switch it to default #4215 [Martin]
      • ESCAPE already testing this quite successfully
      • Testing it in ATLAS, starting in 2 weeks
    • Migrate documentation to new docusaurus [Martin]
    • Client ticket cleanup [Mario]
  • Done
    • Temporarily exclude RSEs with a timeout to not impact general deletion rate #528 [Thomas, Cedric]
• Discussion: Getting rid of oracle triggers?
  • Martin is in favor
  • Mario is in favor
  • Check with Oracle people at CERN about triggers
    • If nothing speaks against it, remove them with 1.25/1.26
• Full stack upload/download/transfer testing [Mayank]
  • xrootd protocol upload/download tests PR exists (Needs review!)
    • Ready to merge; Change with uploadclient can be submitted later
  • TPC PR
    • Should be ready soon for review
• Parallel testing [Ben]
• Documentation
  • Ongoing section to collect documentation issues

15:55 → 16:00 AOB