Rucio Development Meeting

Thursday 15 Aug 2019, 15:00 → 16:30 Europe/Zurich

4/S-056 (CERN)

Martin Barisits (CERN)

15:00 → 15:10 News

• Release schedule
  • Final 1.20.4 to be released on 09.08.2019
    • But... we're still on 1.20.4rc3
  • Original target for 1.20.5 was 26.08.2019
    • Most likely will be pushed back
• New development container
  • With PostgreSQL, FTS, 3*xrootd
  • https://github.com/rucio/rucio/tree/master/etc/docker/dev
  • Ready to use already now, some improvements still coming:
    • performance tuning (also osmosis into travis container)
    • real TPC between xrootd (some outstanding auth problems)
    • even more docs
  • Target
    • have a fully working dev environment with all dependencies (no more mock implementations)
    • get rid of the demo container
    • base of development for the coding camp

15:10 → 15:30 News from the experiments

• Multi VO
  • All the tests pass!
  • Some daemons still to finish, looking very good
  • Ian writing docs for Multi-VO
    • Plan is to deploy test in September/October
    • Hopefully be container-based ;-)
• James & Rohini for ESCAPE / SKAO
  • Working through the development environment
  • Problems with performance of unit tests
• James also has |Travis performance/notifications problems
  • S3 support/storage for dev container
• Riccardo working Aris in ESCAPE
  • Xcache support
  • Integration with Rucio
• Natalia
  • new hire: Fabio
  • assign quotas to users
  • synchronisation of CRIC -> Rucio
  • quotas in CRIC as well
    • CMS policy most likely more complex than ATLAS
  • monitoring: prometheus instead of graphite
• Aris
  • no news yet on import/export CRIC -> Rucio ... still setting up dev environment
  • --> #escape or #cric channel on slack
  • (Panos on holidays)
  • Metadata populated 1 mio DIDs then 10 mio DIDs
    • 0.3 seconds -> 1 mio, 2+ seconds on 10 mio
    • continue with metadata tests
    • later then with MySQL and Oracle
    • DBOD blew up ;-)
    • Problem with undertaker on deletion with foreign key constraint (?)--> bug
• WebUI / Tracking of DID (Hannes)
• SWAN Notebook to analyse Rucio data
  • Hadoop + Spark
• ATLAS Ops is running well :-)
• Eric
  • Hermes open connections to the broker
  • Have to check why connections are kept open

15:30 → 15:50 Hot topics

• Oracle19c testing
  • Currently being set up by Gancho&Co on INT8R
  • Full copy of ATLAS data without history (3.7TB in total)

15:50 → 16:20 Developers roundtable

• Rucio 1.21 priority followup
  • Focus
    • Cleanup & Stability
    • Documentation
    • Deployment (Kubernetes!)
  • Open ID connect #2612
    • Will use refresh tokens to store in the database
    • Need to setup auth server for testing
    • --> Possibility to test with a pre release for ESCAPE?
    • Lifetime of refresh token is infinite at the moment
  • Rucio.cfg vs Rucio config table cleanup #2630
    • Will start a document and have every component responsible to comment;
  • Documentation for configuration parameters #2631
  • History table definition explicitly (No Versioned models) #2063
  • MultiVO features #2635
  • Reaper 2.0 #2412
    • PR for the first prototype is there
      • Already being tested on small amount sites
    • Relax protection of sources #1637
      • More complicated than we thought;
      • Query should scale but need to see if it will hold under heavy load;
    • Fill contents history when deleting last replica #37
    • Archive Handling in Reaper #1431
      • Current issue we discussed in Community news
    • Reaper protection in case of Judge-Evaluator backlog #1578
  • Operators documentation / recipes #2636
  • Expand Kubernetes usage
    • conveyor-submitter, poller, finisher and undertaker are now running
      • Running stable now
      • Almost all daemons are running now on kubernetes
      • hermes and reaper
  • Tracking what happened with a did #2637
    • In progress; Some code from the coding camp, but needs to be continued
  • XCache config table population add to probe #2638
    • Needs to be commited
  • BB8 Needs better configuration and get rid of hard-coded entries
  • Better way to deal with configuration/permissions (entry point, configuration.py, …) #533
  • Transparent handling of archives with rules #1091
  • Global Quotas #2315
    • Making progress; Updating the CLI
  • Possibility to inject rules delayed #2639
  • Improve oracle test crashes #2588
    • Some improvements, unclear if it helped
  • Python 3.5 for server
    • pystatsd does not support Python3
      • Change to statsd possible
    • Postgres issue with byte and text data
  • Python 3.6 for clients
    • Should be compatible, but something missing in the tests
  • Changes for CTA transfer handling #2632
    • Non trivial, still needs to be discussed
    • Could do multi-hop handling right away with this, but should focus development on CTA handling first
  • Source throttling #2611
    • Almost done; Need to add src_rse_id
    • Now using rse_transfer limits table
    • Will need schema change as well
    • One test failing because of sqlalchemy with sqlite; Otherwise development is done
  • Activity exclusion for submitter #2640

16:20 → 16:30 AOB

• Next meeting Aug 29: https://indico.cern.ch/event/833371/