Rucio Development Meeting

Thursday 1 Aug 2019, 15:00 → 16:30 Europe/Zurich

4/S-056 (CERN)

Martin Barisits (CERN)

15:00 → 15:10 News

• Release schedule
  • 1.20.4rc1 on 02.08.2019 (unusual date)
    • Testing the ScopeName PR for Multi-VO Rucio
    • Final 1.20.4 to be released on 09.08.2019
  • 1.20.5 on 26.08.2019

15:10 → 15:30 News from the experiments

• ATLAS
  • Some issues with judge-repairer
    • Ressource consumption
  • New reaper in test mode
    • Query can be improved
    • All tier=3 sites and small tier=2 sites
• ESCAPE
  • New fellows in ESCAPE - Welcome Richard, Ricardo!
    • Involved in transfer related projects
      • Sonar/Noted/Large scale data transfers (http/xrootd)
      • Token Based authentication
    • XCache and Caches in Rucio
• CMS
  • Million File test
    • 630k files
    • 98% level
    • 300TB
  • Setup of reaper2
    • Included in helm-chart
  • Oracle19c test as well
• RAL Multi-VO
  • Most of the functionality there and working for the test
• DUNE/Objectstores
  • Add additional VO policies

15:30 → 15:50 Hot topics

15:50 → 16:20 Developers roundtable

• Rucio 1.21 priority followup
  • Focus
    • Cleanup & Stability
    • Documentation
    • Deployment (Kubernetes!)
  • Open ID connect #2612
    • Will use refresh tokens to store in the database
    • Need to setup auth server for testing
    • --> Possibility to test with a pre release for ESCAPE?
    • Lifetime of refresh token is infinite at the moment
  • Rucio.cfg vs Rucio config table cleanup #2630
    • Will start a document and have every component responsible to comment;
  • Documentation for configuration parameters #2631
  • History table definition explicitly (No Versioned models) #2063
  • MultiVO features #2635
  • Reaper 2.0 #2412
    • PR for the first prototype is there
      • Already being tested on small amount sites
    • Relax protection of sources #1637
      • More complicated than we thought;
      • Query should scale but need to see if it will hold under heavy load;
    • Fill contents history when deleting last replica #37
    • Archive Handling in Reaper #1431
      • Current issue we discussed in Community news
    • Reaper protection in case of Judge-Evaluator backlog #1578
  • Operators documentation / recipes #2636
  • Expand Kubernetes usage
    • conveyor-submitter, poller, finisher and undertaker are now running
      • Running stable now
      • Almost all daemons are running now on kubernetes
      • hermes and reaper
  • Tracking what happened with a did #2637
    • In progress; Some code from the coding camp, but needs to be continued
  • XCache config table population add to probe #2638
    • Needs to be commited
  • BB8 Needs better configuration and get rid of hard-coded entries
  • Better way to deal with configuration/permissions (entry point, configuration.py, …) #533
  • Transparent handling of archives with rules #1091
  • Global Quotas #2315
    • Making progress; Updating the CLI
  • Possibility to inject rules delayed #2639
  • Improve oracle test crashes #2588
    • Some improvements, unclear if it helped
  • Python 3.5 for server
    • pystatsd does not support Python3
      • Change to statsd possible
    • Postgres issue with byte and text data
  • Python 3.6 for clients
    • Should be compatible, but something missing in the tests
  • Changes for CTA transfer handling #2632
    • Non trivial, still needs to be discussed
    • Could do multi-hop handling right away with this, but should focus development on CTA handling first
  • Source throttling #2611
    • Almost done; Need to add src_rse_id
    • Now using rse_transfer limits table
    • Will need schema change as well
    • One test failing because of sqlalchemy with sqlite; Otherwise development is done
  • Activity exclusion for submitter #2640

16:20 → 16:30 AOB

• Next meeting in 2 weeks