Rucio Development Meeting

Thursday 14 Jan 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• Reminder vCHEP
• Reminder: Discussion of GSOC projects in meeting next week
• Release schedule
  • 1.24.1.post1 today
  • 1.24.2 next week
  • 1.24.3 either in 1/2 weeks
• PaN ESCAPE Data Management Workshop

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • 1.24.1.post1 fix for the poller
  • Open Data (ATLAS & CMS Rucio)
    • Possible use Rucio for transfers?
• CMS
  • No news, still on 1.23.post3
• Fermilab
  • Need to check with small experiments for a roadmap for 2021
• Belle II
  • Started today with the migration
    • Draining at the moment
    • LFC Import in 7h
    • Weekend: checking
  • Python issue which was fixed
• LDMX
  • Will upgrade to 1.24 in the next weeks
• DUNE/Edinburgh
  • Lightweight clients
  • Write eMail to rucio-dev to get everyone informed
• Multi-VO
• ESCAPE
  • Some issues with Judge daemons
  • Some problems with FTS?
  • ESCAPE FDR results presented at GDB yesterday

15:20 → 15:30 Hot topics

15:30 → 15:55 Developers roundtable

• Rucio 1.25 "Rat-Donkey" release followup
  • In Progress
    • Stronger integration of Globus Online transfertool #4216 [Matt, Ben]
      • New transfertool column in requests table (To select the right transfertool a-priori)
      • Conveyor-preparer is filling this information
      • Submitter will be started with specific transfertool (FTS3, GO)
      • Code aimed to be done by Jan-22, then testing
      • Ben completed most of the work in the preparer
        • Can be reviewed already
        • Logging function has been added as well for easier daemon logging
    • rucio.cfg vs config table #2630 [Mario]
      • Unit tests needs some work, otherwise done
    • Remove webpy endpoints and dependency #4044 [Ben]
      • Can already start testing FLASK backend
        • PR for containers
        • On puppet hosts can change WSGI endpoints
        • Lets switch ATLAS int servers to FLASK on Wednesday
      • Looking into loading of FLASK endpoints
    • Quality of Service #3419 [Mario, Martin, *]
    • Deprecate reaper1 #4213 [Martin]
      • Should be done by next week, still some issues with testcases
  • Todo
    • Test new rule mode and switch it to default #4215 [Martin]
      • ESCAPE already testing this quite successfully
      • Testing it in ATLAS, starting in 2 weeks
    • Migrate documentation to new docusaurus [Martin]
    • Client ticket cleanup [Mario]
    • Versioned (History) Tables should be defined explicitly #2063 [Martin]
    • Temporarily exclude RSEs with a timeout to not impact general deletion rate #528 [Thomas, Cedric]
    • Logging review #4220 [Component Leads]
      • Internal deadline: Feb-11
    • Identify and cleanup unused functionality and code #4221 [Component Leads]
      • Internal deadline: Feb-11
      • pytest coverage - need to try
  • Done
• Discussion: Getting rid of oracle triggers?
  • Martin is in favor
  • Mario is in favor
  • Check with Oracle people at CERN about triggers
    • If nothing speaks against it, remove them with 1.25/1.26
• Full stack upload/download/transfer testing [Mayank]
  • xrootd protocol upload/download tests PR exists (Needs review!)
    • Finishing up integration of Bens comments
    • Runtime 5-6 minutes for everything
  • TPC PR
    • Will submit draft PR for this
• Parallel testing [Ben]
  • xdist python distributed testing
  • Mark every test which is not able to be called in parallel
  • Also mark tests which leave leftovers behind (The test should probably clean this up itself)
  • Good news!
    • Non-parallel run of tests is working
    • Currently looking through tests which have to be defined as non-parallel

15:55 → 16:00 AOB

• Discussion: Rucio Coding Camp 2021
  • Remote Hackathon, Mayank has some
  • Dedicated list of topics, then work on specific ones
  • Full day or Half-Day?
    • Timezone difference could be problematic