Rucio Development Meeting

Thursday 15 Apr 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:10 News

• Component lead changes due to recent and upcoming departures
  • Clients: Mario, Tomas, Radu
  • Consistency: Tomas, Dimitrios
  • Docker: Thomas, Mario, Radu
  • Monitoring: Thomas, Mario, Radu
  • Protocols: Mario, Tomas, Radu
  • Rebalancing: Tomas, Cedric
  • Recovery: Cedric, Tomas
  • REST & API: Ben, Martin
  • Traces: Thomas, Mario
  • Transfers: Martin, Cedric, Eric, Radu
  • WebUI: Thomas, Martin, Cedric​​​​​​​
• ​​​​​​​Next week: Hermes 2 planning & discussion

15:10 → 15:20 Community News & DevOps roundtable

• ATLAS
  • ActiveMQ changes need to be applied
  • Issue with poller on Kubernetes and memory limit
    • Large amount of Staging requests which are very large, went slightly over memory limit
      • Increasing limit solved it
• Belle II
  • Finally got WebUI running
    • Issue was that the WebUI did not have access to the DB
      • Needs access to check token
      • Error was misleading which lead to this difficult debugging
      • Another good reason to get rid of webui accessing the DB (Just redirecting)
        • In the meanwhile, better error message would already help
    • Some views are empty which rely on dumps
      • No real documentation for that
      • Subscription table (Showing all subscriptions) relies on a dump
        • Better move to REST endpoints and caching?
  • Metadata
    • Testing import of all Belle II metadata into Rucio (80M files, 10-15 fields per file)
• LDMX
  • No news
• DUNE/Edinburgh
  • LFN2PFN converter for DUNE
  • More lightweight client work soon
• Multi-VO/RAL
  • Setting up some RSEs on the RAL instance
• ESCAPE
  • No News
• SKA
  • No News

15:20 → 15:30 Hot topics

15:30 → 15:55 Developers roundtable

Rucio 1.26 "Donkey League of La Mancha" priority followup

• In Progress
  • Test and switch to new rule algorithm #4215 [Martin]
    • Remove old code for 1.26
  • Migrate documentation to new docusaurus [Martin, Ben, Daniel]
    • man page generation for the doc
      • A bit different compared to sphinx
    • Documentation repo branch setup and running
    • Branch ready for rucio main repo
      • Can be merged soon
  • Logging Review #4220 [All Component leads]
  • Re-Factor Transfer workflow [Radu]
    • Re-Structure PR merged
    • Now reviewing the bigger feature PR 
  • Discuss and plan mysql5 deprecation [Mario]
    • No news from community
    • Remove support with 1.26, re-structure DB calls/migrate script with 1.27
  • Test SSO Login [Rizart]
    • Studied API and look into connection to CERN SSO
  • Test Import and Export of CRIC JSON [Rizart]
    • ESCAPE needs custom parameters to add to RSEs 
      • Backported feature from ATLAS plugin to mainstream CRIC
      • Some fields in the RSE table, some attributes
      • Discuss with CRIC people to have a more consolidated solution
  • Auditor overhaul #3437 [Dimitrios, Eric] 
  • Hermes2 planning
• Todo
  • Switch to policy-packages as default [James, Martin]
  • Versioned history tables #2063 [Martin]
  • Rename reaper2 to reaper #4462 [Martin]
  • rucio.cfg vs Config table #2630 [Mario] 
  • Python 3.8 and 3.9 testing #4453 [Ben]
  • Remove webpy endpoints and dependency #4044 [Ben]
  • Setup.py and requirements overhaul #4456 [Ben]
  • Cleanup tickets #4463 [*]
• Done
• Delayed
  • QoS #3419 [Mario, Martin, Rizart]

Developer roundtable

• Integration testing
  • FTS container not doing transfers, needs some work
• Testing issue with Deadlocks
  • Radu made 2 PRs to fix some of these

15:55 → 16:00 AOB