Rucio Development Meeting

Thursday 29 Aug 2019, 15:00 → 16:30 Europe/Zurich

304/1-001 (CERN)

Martin Barisits (CERN)

15:00 → 15:10 News

• Release schedule
  • 1.20.5 on Sep 6
  • 1.20.6 on Sep 23
  • 1.20.7 on Oct 14
  • (Oct 15 - Oct 17 Rucio coding camp)
  • 1.21 "Donkeys of the Galaxy" end October/begin November
• Rucio journal paper released
• • https://link.springer.com/article/10.1007/s41781-019-0026-3
  • Open Access
• Meeting schedule
  • September we resume the weekly development meeting schedule
  • Sep 5 NO MEETING (CERN holiday)

15:10 → 15:30 News from the experiments

• ATLAS
  • Summer holidays!
• CMS
  • Rucio data export
  • Distance export issue
    • Will be tracked down
  • Release candidate build of docker containers does not work - builds the latest non RC version
• DUNE/Edinburgh
  • Two open PRs: Object stores and Policy/Permissions
• ESCAPE
  • Import/Export
    • accounts, quotas did not work
  • Generic metadata
    • Benchmark on postgres and mysql
      • postgres for 10mio rows finding a row took 2s
      • mysql takes ~2m
      • Oracle tests will come soon
    • Undertaker bug found
      • Different schema with dev container than production version?
      • Need to identify this

15:30 → 15:50 Hot topics

15:50 → 16:20 Developers roundtable

• Rucio 1.21 priority followup
  • Focus
    • Cleanup & Stability
    • Documentation
    • Deployment (Kubernetes!)
  • Open ID connect #2612
    • Provisioning client used similar to query VOMS (--> probes)
    • Will need a release candidate to test with e.g. ESCAPE
  • Rucio.cfg vs Rucio config table cleanup #2630
    • Will start a document and have every component responsible to comment;
  • Documentation for configuration parameters #2631
  • History table definition explicitly (No Versioned models) #2063
  • MultiVO features #2635
  • Reaper 2.0 #2412
    • PR for the first prototype is there
      • Already being tested on small amount sites
    • Relax protection of sources #1637
      • More complicated than we thought;
      • Query should scale but need to see if it will hold under heavy load;
    • Fill contents history when deleting last replica #37
    • Archive Handling in Reaper #1431
      • Current issue we discussed in Community news
    • Reaper protection in case of Judge-Evaluator backlog #1578
  • Operators documentation / recipes #2636
  • Expand Kubernetes usage
    • conveyor-submitter, poller, finisher and undertaker are now running
      • Running stable now
      • Almost all daemons are running now on kubernetes
      • hermes and reaper
    • All daemons running now on Kubernetes
    • Should add servers
    • Plan is to move from ATLAS integration puppet to kubernetes
  • Tracking what happened with a did #2637
    • In progress; Some code from the coding camp, but needs to be continued
  • XCache config table population add to probe #2638
    • Needs to be commited
  • BB8 Needs better configuration and get rid of hard-coded entries
  • Better way to deal with configuration/permissions (entry point, configuration.py, …) #533
  • Transparent handling of archives with rules #1091
  • Global Quotas #2315
    • Making progress; Updating the CLI
    • PR there, needs review
  • Possibility to inject rules delayed #2639
  • Improve oracle test crashes #2588
    • Some improvements, unclear if it helped
  • Python 3.5 for server
    • pystatsd does not support Python3
      • Change to statsd possible
    • Postgres issue with byte and text data
  • Python 3.6 for clients
    • Should be compatible, but something missing in the tests
  • Changes for CTA transfer handling #2632
    • Non trivial, still needs to be discussed
    • Could do multi-hop handling right away with this, but should focus development on CTA handling first
  • Source throttling #2611
    • Almost done; Need to add src_rse_id
    • Now using rse_transfer limits table
    • Will need schema change as well
    • One test failing because of sqlalchemy with sqlite; Otherwise development is done
  • Activity exclusion for submitter #2640​​​​​​​

16:20 → 16:30 AOB

• No meeting next week