Rucio Development Meeting

Thursday 21 Jan 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• February Meeting schedule
  • Feb 4
  • Feb 11
  • Feb 18
  • Feb 25
• Release schedule
  • 1.24.1.post2 still having an issue with state strings
    • Potentially confusing some monitoring
    • 1.24.1.post3 will still come today
  • 1.24.2 next
  • 1.24.3 in 2 weeks
• Thanks for submitting content for the Rucio Community vCHEP paper

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • 1.24.1.post2 deployed and found another bug
    • 1.24.1.post3 coming soon
  • Move to python3
    • Puppet hosts all moved to python3 as well
• Belle II
  • Migration done and Belle II running on Rucio now
    • Some simple fix had to be applied
  • Noticed some memory leak on the server
    • In 2 days consumed all the memory
• CMS
  • No news
• VIRGO/LIGO
  • Definitions of roles in computing policy
• STFC/MultiVO
  • Nothing to report
• ESCAPE
  • 1.24 messaging issue seen as well
  • With 1.23 deletion did not properly succeed, with 1.24 seems to work?
  • Seeing still some restarts from judge-eval
    • Probably due to oracle instaclient version

15:20 → 15:30 Hot topics

• GSOC Rucio Projets
  • Upload/Download [Mario]
    • rclone as an alternative to gfal copy
  • Probes to daemons conversion [Eric]
    • Will discuss offline if suitable for GSOC
  • Contribution "staring-ideas" as an idea pool

15:30 → 15:55 Developers roundtable

• Rucio 1.25 "Rat-Donkey" release followup
  • In Progress
    • Stronger integration of Globus Online transfertool #4216 [Matt, Ben]
      • New transfertool column in requests table (To select the right transfertool a-priori)
      • Conveyor-preparer is filling this information
      • Submitter will be started with specific transfertool (FTS3, GO)
      • Code aimed to be done by Jan-22, then testing
      • Ben completed most of the work in the preparer
        • Can be reviewed already
        • Logging function has been added as well for easier daemon logging
      • Both PRs (Preparer + Submitter + Poller) submitted
        • Needs a bit more documentation
        • Deployment we need to see for testing
    • rucio.cfg vs config table #2630 [Mario]
      • Unit tests needs some work
      • After S&C week
    • Remove webpy endpoints and dependency #4044 [Ben]
      • Can already start testing FLASK backend
        • PR for containers
        • On puppet hosts can change WSGI endpoints
        • Lets switch ATLAS int servers to FLASK on Wednesday
      • Looking into loading of FLASK endpoints
      • PR for configuration change
      • Start using FLASK on integration server of ATLAS next week
    • Quality of Service #3419 [Mario, Martin, *]
    • Deprecate reaper1 #4213 [Martin]
      • Should be done by next week, still some issues with testcases
      • Reaper2 testcase exposed a few differences to reaper1
        • MaxFilesBeingDeleted setting does not work anymore -- remove?
        • Deleting until exactly the threshold does not work in reaper2
    • Temporarily exclude RSEs with a timeout to not impact general deletion rate #528 [Thomas, Cedric]
      • add config_get (file/table)
    • Logging review #4220 [Component Leads]
      • Internal deadline: Feb-11
    • Identify and cleanup unused functionality and code #4221 [Component Leads]
      • Internal deadline: Feb-11
      • pytest coverage - need to try
    • Versioned (History) Tables should be defined explicitly #2063 [Martin]
  • Todo
    • Test new rule mode and switch it to default #4215 [Martin]
      • ESCAPE already testing this quite successfully
      • Testing it in ATLAS, starting in 2 weeks
    • Migrate documentation to new docusaurus [Martin]
    • Client ticket cleanup [Mario]
  • Done
• Discussion: Getting rid of oracle triggers?
  • Martin is in favor
  • Mario is in favor
  • Check with Oracle people at CERN about triggers
    • If nothing speaks against it, remove them with 1.25/1.26
• Full stack upload/download/transfer testing [Mayank]
  • xrootd protocol upload/download tests PR exists (Needs review!)
    • Finishing up integration of Bens comments
    • Runtime 5-6 minutes for everything
  • TPC PR
    • Will submit draft PR for this
  • Further work on Upload/Download PR
  • Testing: TPC webdav Upload/Download xroot
• Parallel testing [Ben]
  • xdist python distributed testing
  • Mark every test which is not able to be called in parallel
  • Also mark tests which leave leftovers behind (The test should probably clean this up itself)
  • Good news!
    • Non-parallel run of tests is working
    • Currently looking through tests which have to be defined as non-parallel

15:55 → 16:00 AOB