Rucio Development Meeting

Thursday 29 Apr 2021, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• May meeting schedule
  • May-6
  • May-13 Holiday in Geneva
  • May-20 vCHEP parallel sessions
  • May-27

15:05 → 15:15 Community News & DevOps roundtable

• ATLAS
  • ATLAS Local Rootbase
    • Switch to Python3.7 for Rucio clients (Dual stack deployment, default for wrapper is py3, for non-wrapper deployment still py2)
      • Need discussion for Python2 client-support continuation
      • Ideally to discuss at the Rucio Community Workshop
• CMS
  • Setup GO transfers for CMS
    • Trying to setup infrastructure
    • Ask Matt to extend the documentation for rse_settings for GO
• Fermilab
  • Why grafana monitoring is not worning
    • No data ever shows up
• Belle II
  • Metadata ingesting
  • Plenary talk at vCHEP for Belle Rucio integration
  • Working on webUI
    • Trying to setup the dumps
    • Dump only works on CEST, not other timezones --> Should be switched to UTC
• Regensburg Lattice QCD
  • Waiting to setup new test env
• LDMX
• RAL / MultiVO
  • Transfers with MultiVO not working
  • All transfers being done under one certificate
    • Needs VO - Certificate mapping in conveyor
      • Possible issue with grouping method
• ESCAPE
  • Infrastructure updates
  • Auth server + openid connect
    • Redirection back from IAM to auth server does not work
    • Possibly ingress controller on K8s preventing it
• SKAO
  • Data ingestion tests
  • Policy packages
    • James doc page describing how to write/integrate it
  • GO

15:15 → 15:35 Hermes2 evolution

Speakers: Cedric Serfon (Brookhaven National Laboratory (US)), Mario Lassnig (CERN), Dr Riccardo Di Maria (CERN), Rizart Dona (CERN), Martin Barisits (CERN)

Slides

15:35 → 15:55 Developers roundtable

Rucio 1.26 "Donkey League of La Mancha" priority followup

• In Progress
  • Test and switch to new rule algorithm #4215 [Martin]
    • Remove old code for 1.26
  • Migrate documentation to new docusaurus [Martin, Ben, Daniel]
    • PR is ready to remove doc/ folder
    • Need to add it then to documentation repo
    • Some missing changes
      • Release notes
      • Some other minor changes
    • Deprecation of readthedocs
      • Maybe add a header about the new documentation but leave the old documentation intact
      • Freezing it, but leaving it until full deprecation
  • Logging Review #4220 [All Component leads]
  • Re-Factor Transfer workflow [Radu]
    • Re-Structure PR merged
    • Now reviewing the bigger feature PR
  • Discuss and plan mysql5 deprecation [Mario]
    • No news from community
    • Remove support with 1.26, re-structure DB calls/migrate script with 1.27
    • Need to remove tests for mysql5 and do a historic cleanup of alembic
  • Test SSO Login [Rizart]
    • Studied API and look into connection to CERN SSO
  • Test Import and Export of CRIC JSON [Rizart]
    • ESCAPE needs custom parameters to add to RSEs 
      • Backported feature from ATLAS plugin to mainstream CRIC
      • Some fields in the RSE table, some attributes
      • Discuss with CRIC people to have a more consolidated solution
      • Checked how the different plugins are differentiated
        • Will possibly need a plugin for each community
    • ESCAPE already uses the rucio importer (full json import)
  • Auditor overhaul #3437 [Dimitrios, Eric] 
  • Hermes2 planning
    • Postponed to next week
  • Remove webpy endpoints and dependency #4044 [Ben]
    • Needs container repo PR first, to switch dev container to FLASK
• Todo
  • Switch to policy-packages as default [James, Martin]
  • Versioned history tables #2063 [Martin]
  • Rename reaper2 to reaper #4462 [Martin]
  • rucio.cfg vs Config table #2630 [Mario] 
  • Python 3.8 and 3.9 testing #4453 [Ben]
  • Setup.py and requirements overhaul #4456 [Ben]
  • Cleanup tickets #4463 [*]
• Done
• Delayed
  • QoS #3419 [Mario, Martin, Rizart]

Developer roundtable

• Integrating Belle DIRAC Rucio solution into Multi-VO RAL rucio server
  • Rucio clients are delivered with DIRAC workload
  • Goal is to use DIRAC Configuration to configure Rucio
  • Right now rucio clients strictly require a config file
    • $RUCIO_HOME can specify directory where rucio.cfg is located
  • Currently import rucio clients
    • logging is setup on input level
  • Not loading rucio.cfg but read config from ENV instead?
  • Issue is that during import of the client already setup_logger is called, which crashes without config
    • Followup offline
    • Problem might extend outside the setup_logger aswell
• Integration testing
  • Change requests all implemented
  • Changed pytest plugin
  • Should work now
•  

15:55 → 16:00 AOB