Rucio Development Meeting

Thursday 22 Apr 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• Hermes2 evolution
  • Discussion scheduled for today but due to meeting conflicts, postponed to next week
• Rucio CHEP Slides
  • @authors: Please add your slides
• Rucio Community Workshop
  • Program committee had first meeting
  • First announcement will be sent out soon
• 1.25.3.post1 hotfix release
  • Due to regression in hermes/messaging

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • More ATLAS K8s stuff
• CMS
  • Setup integration instance for testing
    • On 1.25 now
• LDMX
• DUNE/Edinburgh
  • More LFN2PFN work
    • Integrating with the DUNE policy package
  • Lightweight client
    • Make the client run without config file, but run only with ENV variables
      • Yes, general interest in this
    • Will also have CERN Summer student to work on this
• MultiVO/STFC
  • New RSEs integrated
    • Some issues to be fixed
• ESCAPE
  • Move to flux for K8s config management
• SKA
  • Stabilising the deployment
  • CRON job which dumps db to be able to restore if needed
  • Re-Configuring some of the RSEs
  • Data ingestion tests
  • Next: Probes
    • Are they commonly usable or still ATLAS specific?
      • Generally usable, still a few ATLAS specific ones (If you see ATLAS mentioned...)
      • Splitting the probes into common and community specific ones is still ongoing, but should be finished soon
    • Can you run them on K8s?
      • Yes

15:20 → 15:40 Hermes2 evolution (Postponed to next week)

15:40 → 15:55 Developers roundtable

Rucio 1.26 "Donkey League of La Mancha" priority followup

• In Progress
  • Test and switch to new rule algorithm #4215 [Martin]
    • Remove old code for 1.26
  • Migrate documentation to new docusaurus [Martin, Ben, Daniel]
    • PR is ready to remove doc/ folder
    • Need to add it then to documentation repo
    • Some missing changes
      • Release notes
      • Some other minor changes
    • Deprecation of readthedocs
      • Maybe add a header about the new documentation but leave the old documentation intact
      • Freezing it, but leaving it until full deprecation
  • Logging Review #4220 [All Component leads]
  • Re-Factor Transfer workflow [Radu]
    • Re-Structure PR merged
    • Now reviewing the bigger feature PR
  • Discuss and plan mysql5 deprecation [Mario]
    • No news from community
    • Remove support with 1.26, re-structure DB calls/migrate script with 1.27
    • Need to remove tests for mysql5 and do a historic cleanup of alembic
  • Test SSO Login [Rizart]
    • Studied API and look into connection to CERN SSO
  • Test Import and Export of CRIC JSON [Rizart]
    • ESCAPE needs custom parameters to add to RSEs 
      • Backported feature from ATLAS plugin to mainstream CRIC
      • Some fields in the RSE table, some attributes
      • Discuss with CRIC people to have a more consolidated solution
      • Checked how the different plugins are differentiated
        • Will possibly need a plugin for each community
    • ESCAPE already uses the rucio importer (full json import)
  • Auditor overhaul #3437 [Dimitrios, Eric] 
  • Hermes2 planning
    • Postponed to next week
  • Remove webpy endpoints and dependency #4044 [Ben]
    • Needs container repo PR first, to switch dev container to FLASK
• Todo
  • Switch to policy-packages as default [James, Martin]
  • Versioned history tables #2063 [Martin]
  • Rename reaper2 to reaper #4462 [Martin]
  • rucio.cfg vs Config table #2630 [Mario] 
  • Python 3.8 and 3.9 testing #4453 [Ben]
  • Setup.py and requirements overhaul #4456 [Ben]
  • Cleanup tickets #4463 [*]
• Done
• Delayed
  • QoS #3419 [Mario, Martin, Rizart]

Developer roundtable

• Integration testing
  • Issue with FTS transfers has been fixed 
    • PR is submitted for that
      • Using python2 to activate RSEs, delegate proxy
    • PR on containers repo to setup pip2 for fts-rest
  • TPC PR
    • Use fts rest CLI to validate actions of Rucio daemons
    • Need to move it to a GH action now
• Issue with Daemons using the wrong DN
  • Lookup table to map VO to DNs
•  

15:55 → 16:00 AOB