Rucio Development Meeting

Thursday 25 Feb 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• March meeting schedule
  • Mar 4
  • Mar 11
  • Mar 18
  • Mar 25
• 1.25 Code Freeze
  • In principle we are in code-freeze
  • Still need to do some merging
  • Please let me know about important PR you would prefer in 1.25.0
• Releases
  • Several releases this week (1 Patch, 2 LTS)
  • Good feedback for LTS releases
• CHEP Paper
  • Good to go
  • Still some additions (ESCAPE and possible VIRGO/LIGO)
  • Will add the latest version to CDS

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Logging object from pilot to downloadclient broke pilot
    • Pilot passes logging object, we expect logging function
    • Switched back to object in next release
  • Moved last few puppet nodes to py3
    • redirect and ui
    • auth still suffering from problem with import
      • PR for this submitted
  • Removed more daemon threads from puppet
    • Now 90%+ of the workload is on kubernetes
  • New rule mode being evaluated at the moment
    • Will have two rounds of testing, then switch completely
• Fermilab
  • No news
  • Still checking differences with clusters
• Belle II
  • Struggling with deployment of webui 
    • Requires proxy in front of any outside exposed nodes
    • Client certificate is not passed through to UI node
  • Increased volume by 10%
  • DIRAC plugin working
• LDMX
  • Put set rse_limits/usage into rucio-admin CLI
  • Generic permission file too restrictive
• RAL / Multi VO
  • Patching infrastructure of Rucio cluster at RAL
  • VO Name length
• ESCAPE / SKAO
  • Setting up an SKA Rucio instance
  • Set up a few daemons, WebUI not yet
  • Also added some storages 
  • RSE Usage / Limits difficult to folow
  • Tutorials have been very helpful + CMS repo and howtos

15:20 → 15:30 Hot topics

mypy.pdf

• mypy presentation from Ben

15:30 → 15:55 Developers roundtable

• Rucio 1.25 "Rat-Donkey" release followup
  • In Progress
    • Logging review #4220 [Component Leads]
      • Internal deadline: Feb-11 (passed)
      • still ongoing
      • New loggers all added
        • Setup is there, some daemons are missing the prefix (heartbeat)
      • PLEASE go through your components and cleanup log messages
    • Versioned (History) Tables should be defined explicitly #2063 [Martin]
      • Should be in 1.25
    • Client ticket cleanup [Mario, Radu]
    • •  
  • Todo
    • Migrate documentation to new docusaurus [Martin]
      • Bring in sync readthedocs docu to new rucio/documentation, Daniel will help
      • API documentation generation does not work
      • Daniel will look into it this week
  • Done
    • Stronger integration of Globus Online transfertool #4216 [Matt, Ben]
      • PR is ready, Martin need to review
      • Need to add documentation to describe how to run GO with Rucio
    • Deprecate reaper1 #4213 [Martin]
      • Submit this afternoon
  • Delayed
    • Quality of Service #3419 [Mario, Martin, Rizart, *]
    • rucio.cfg vs config table #2630 [Mario]
    • Remove webpy endpoints and dependency #4044 [Ben]
      • FLASK endpoints running on ATLAS integration
        • Needs more testing (Authentication, Traces, WebUI)
      • Will leave web.py in 1.25 --> aimed to 1.26
    • Identify and cleanup unused functionality and code #4221 [Component Leads]
    • Test new rule mode and switch it to default #4215 [Martin]
      • ESCAPE already testing this quite successfully
      • Testing it in ATLAS - activate next week
        • Will be needed, otherwise won't run on K8s
• Full stack upload/download/transfer testing [Mayank]
  • PR for upload/download testing merged
    • RSEManager.upload should be replaced by UploadClient
  • TPC Testing
    • Works, but need to check if it properly works for mock RSEs
• Parallel testing [Ben]
  • Resumed work on it
  • 1/3 through sourcecode for marking parallel/sequencial tests
  • Followup PR to fix the easy tests which should be fixed
• Documentation [Daniel, Martin]
  • Move k8s tutorial into main doc, need to sort out where exactly
  • Extending docs with more details
  • Add "What is Rucio tab" to sidebar on webpage?
    • Martin or Mario can write something
  • Add new documentation content already to new rucio/documentation
  • Move k8s-demo to rucio/ repository
    • Martin will create rucio/rucio-k8s-tutorial
    • Thomas will push repo there
    • DONE
  • Move release-notes to separate directory

15:55 → 16:00 AOB