Rucio Development Meeting

Thursday 11 Feb 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• CERN Summer Student Submission
  • Develop support for Apache/WebDAV and nginx/WebDAV
  • Integration of caches in ESCAPE stack + related monitoring + stress tests of lake
• Issues with 1.24.3
  • reaper
    • applied fix, worked
  • finisher
    • ok for escape now
  • recommend to hotfix
• Release schedule
  • 1.24.4 Feb-15
  • 1.24.5 Feb-22
  • Code Freeze from Feb-24
    • 1.25.0rc1 Feb-25
    • 1.25.0 Mar-1 ... Mar-5

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • 1.24.3 woes
  • investigate lack of notifications for new rules
  • k8s now running with flask instead of web.py
  • next week resiliency test check-on-tape feature of FTS on CTA
• CMS
  • nothing to report
• FNAL
• Belle II
  • setting up rucio-ui
  • defining subscriptions for data-taking for raw data export
    • previously only used BNL, now five data centres
• DUNE/Edinburgh
  • nothing to report
  • removing client dependencies
• ESCAPE
  • 1.24.3 woes
  • deployment of dark-reaper and auditor (with ThomasB, Dimitrios)
    • dark data in lake
  • enable tracing infra + kronos daemon
  • LOFAR
    • try to transfer 58 and 115 GB files to EOS
      • for 58GB: succeeded, even thoug the flow is: first tries gridftp, then times out, then uses xroot, then transfer is cancelled but ends up being successful
      • for 115GB: fails, and errors that the file already exists, but this should not be the case
    • in contact with EOS people, need more investigation
      • Rucio logs exist
    • need more investigation
• VIRGO/LIGO
• STFC/MultiVO
  • interest is increasing
  • minor wrinkles to iron out
  • default permissions need a second pair of eyes
  • restrict to names of
    • three characters too limiting
    • code fix incoming
  • some client changes might be needed for smooth transition
• LDMX
  • nothing to report

15:20 → 15:30 Hot topics

•  

15:30 → 15:55 Developers roundtable

• Rucio 1.25 "Rat-Donkey" release followup
  • In Progress
    • Stronger integration of Globus Online transfertool #4216 [Matt, Ben]
      • PRs done, finalizing tomorrow
    • rucio.cfg vs config table #2630 [Mario]
      • Circular dependency between core-config and common-config
        • Not yet resolved
        • Will need a new module to resolve
    • Remove webpy endpoints and dependency #4044 [Ben]
      • Start using FLASK on integration server of ATLAS next week
        • done
      • Completed OIDC flask template
      • Removal of web.py with 1.25 should still be fine
    • Quality of Service #3419 [Mario, Martin, Rizart, *]
      • still needs development discussions at the lower levels
    • Deprecate reaper1 #4213 [Martin]
      • Should be done by next week, still some issues with testcases
      • Reaper2 testcase exposed a few differences to reaper1
        • MaxFilesBeingDeleted setting does not work anymore -- remove
        • Deleting until exactly the threshold does not work in reaper2
    • Logging review #4220 [Component Leads]
      • Internal deadline: Feb-11 (today :)
      • still ongoing
      • Added decorated logger added to all daemons/server
        • Now prefix string handed through to core layer as well
    • Identify and cleanup unused functionality and code #4221 [Component Leads]
      • Internal deadline: Feb-11
      • Most likely not happening with 1.25
      • pytest coverage - need to try
    • Versioned (History) Tables should be defined explicitly #2063 [Martin]
  • Todo
    • Test new rule mode and switch it to default #4215 [Martin]
      • ESCAPE already testing this quite successfully
      • Testing it in ATLAS, starting in 2 weeks
    • Migrate documentation to new docusaurus [Martin]
    • Client ticket cleanup [Mario]
  • Done
• Discussion: Getting rid of oracle triggers?
  • Martin is in favor
  • Mario is in favor
  • Check with Oracle people at CERN about triggers
    • If nothing speaks against it, remove them with 1.25/1.26
• Full stack upload/download/transfer testing [Mayank]
  • xrootd protocol upload/download tests PR exists (Needs review!)
    • Ready to merge; Change with uploadclient can be submitted later
  • TPC PR
    • Should be ready soon for review
• Parallel testing [Ben]
• Documentation
  • Move k8s tutorial into main doc, need to sort out where exactly
  • Extending docs with more details
  • Add "What is Rucio tab" to sidebar on webpage?

15:55 → 16:00 AOB

• SKA / Rob(?) question: No helm for Hermes2?
  • It's a bit different than Hermes1 (rucio.cfg + configtable)
  • (only used by Belle-II now)
  • Can add to default helm charts
• Some curiosities with the documentation