Rucio Development Meeting

Thursday 2 Apr 2020, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

Video Conferencing: Please join Zoom instead of Vidyo!

Zoom: https://cern.zoom.us/j/413496641

Meeting ID: 413 496 641
Find your local number: https://cern.zoom.us/u/aT2QQfXAo

15:00 → 15:10 News

• Release Schedule
  • 1.22.0 released!
  • 1.22.1 today
  • 1.22.2 on Monday
    • 2 week release cycle afterwards
  • 1.23.0 LTS "The Incredible Donkey" in June
•  

15:10 → 15:20 News from the experiments

• ATLAS
  • Integration with cloud providers
    • Certificate situation complicated
    • Guideline since today how to include cloud resources in WLCG
      • Loadbalancer (run inside cloud) with certificates
    • Some changes in conveyor which currently needs to run against fts-devel
• CMS
• Multi-VO/RAL/STFC
• DUNE
• Belle II
  • Chained subscription mode being implemented
  • Monitoring infrastructure needs to be ramped up

15:20 → 15:30 Hot topics

15:30 → 15:55 Developers roundtable

• 1.23.0 LTS "The Incredible Donkey" priority followup
  • In Progress
    • Documentation overhaul [Martin, Dimitrios]
      • Page Listing config table and RSE Attribute Parameters #2631 [Martin]
      • Operators Documentation and recipe repository #2636 [Martin]
      • Early phase of picking tools/deciding structure/content
        • Separation between generic / VO specific content
      • Possible discussion in 2 weeks for everyone to comment
    • Expand Kubernetes Usage [Thomas]
      • Waiting for Ricardo for node investigation
      • Switch open stack project to multizone clusters again
      • Reaper2 constantly increasing memory usage (until limit is hit) and restarts
        • Confirmed by CMS too
          • ~50 RSEs processed in reaper
        • ATLAS made big jump to 300+ RSEs
        • Being investigated
        • Check memory usage
      • Debug features with attachable containers coming soon
    • AAI/OIDC Testing and Improvements [Jaroslav]
      • Test of propagation of account to transfertool
      • New patch release to deploy the recent developments on WLCG DOMA cluster
    • MultiVO Functionality #2635 [Eli]
      • Bringing work up to date
      • Meeting later on to specify next steps
      • Discussion: Administration of different VOs
        • Securing VOs, Accounts etc.
    • Unification of metadata interfaces #3096 [Aris]
      • PR submitted, waiting for comments
  • To do
    • rucio.cfg vs config table #2630 [Mario]
    • Handling of Archives in the Reaper #1431 [Thomas, Cedric]
    • Log the Parameters used in all POST/PUT requests #2686 [Thomas]
    • New Code management Model #3417[Martin]
    • RSEmgr version 2.0 #3147 [Tomas, Tobi]
    • QoS #3419 [Aris, Mario, Martin]
    • Python 3 #3420 [Martin]
  • Done
• Handling of lost files in archives in the necromancer [Cedric, Tomas]
  • Tomas can look into it
  • Will require additional queries to check for archives
• Auditor discussion [Dimitrios, Tomas]
  • Input 2 files: DB Dump, Storage Dump
  • Can Auditor not directly get DB information from Rucio (instead of relying on DB Dump)?
    • Possible to do both ways?
      • Difficult, since not all information is available in the db for past replica states
  • Auditor compares the 2 states (DB, Storage)
    • Auditor might as well work on DB dump (without generated PFNs) and generate the PFNs during processing
  • pre, common, post actions
    • Directories for DB, Storage dump being filled (externally)
    • Auditor runs and fetches data from the directories
    • Auditor produces output
  • Dimitrios will create a ticket to collect ideas/workflows and we move forward from there
    • Collect usecases there, verify that it works (compared to old auditor)

15:55 → 16:00 AOB