Rucio Development Meeting

Thursday 28 Nov 2019, 15:00 → 16:30 Europe/Zurich

4/S-056 (CERN)

Martin Barisits (CERN)

1 News

• IEEE Technical Committee on Data Engineering contribution
• Releases
  • Two hotfix (post) releases for the 1.21.0 feature release necessary
  • Please take the time and add unit tests to your PRs, these really can avoid these regressions
  • 1.21.1 normal patch release today
• Next week no meeting

2 News from the experiments

• ATLAS
  • New release
• Ian/MultiVO
  • Interest from BNL on MultiVO
  • AAI for WebUI
• James/DUNE/Edinburgh
  • Updated policy package
• Cedric/Belle2
  • Designing a roadmap for the migration right now
  • Should be done soon
• Aris/ESCAPE
  • -

3 Hot topics

4 Developers roundtable

Rucio_Issue_burn_2019-11-28.png

Rucio_priority_progress_2019-11-28.png

• Issue burn update
• Rucio 1.22 "Green Donkey" priority development followup
  • In Progress
    • Kubernetes [Thomas]
      • Plan was to have all integration servers and daemons in K8s by now
      • This is basically done
      • Working on application metrics to get into prometheus
      • First checkbox can basically be ticked
      • Monitoring: Prometheus metrics can also be used in the auto-scaler
    • Reaper 2.0 #2412 [Cedric, Thomas]
      • It's essentially done, a few things need to be improved
      • Query to get list of unlocked replicas needs to be improved
      • Protection has been relaxed, but need to check if it is too much (needs review)
      • Exclude RSEs with timeout: Not yet
      • Archive part: Mario or Martin can maybe help
    • OIDC #2612 [Jaroslav]
    • MultiVO #2635 [Ian, Andrew]
      • Martin will merge this, this or next week
      • Further development when Eli starts
    • Configuration / Permissions / Policy #533 [James]
      • Waiting for comments on latest PR
    • Repository cleanup #5252 [Martin, Thomas]
      • Separate tools to a separate repo 
      • Maybe merge with probes repo?
    • RSEMgr v2.0 #3147 [Tomas, Tobi]
      • Upload function moved to upload client
        • Tests work
      • Rebase will fix conflict
      • We should do upload/download testing with RSEs in Travis
        • Check with [Mario] to get the docker development also into travis
    • Unification of metadata #3096 [Aris]
      • Most of the work should be done by next week
    • General update of BB8 #3148 [Tomas, JJ, Dimitrios]
  • ToDo
    • Data Pre-Placement [Thomas]
    • Cleanup of Issues [Everyone]
      • Update tickets
    • Documentation [Martin]
    • rucio.cfg vs config table #2630 [Mario]
    • Versioned history tables need explicit definition #2063 [Martin]
    • Protection of sources too strict in reaper #1637 [Cedric, Thomas]
    • Add file to content history in reaper #37 [Cedric, Thomas]
    • Reaper should stop in case of judge backlog #1578 [Martin]
    • Handling of Archives in Reaper #1431 [Cedric, Thomas, Mario, Martin]
    • Missing features for Archives #1091 [Mario, Martin, Tobi, Tomas]
    • Documentation page listing config table and RSE attributes #2631 [Mario]
    • Operators doc and repo #2636 [Martin]
• Bulk mode for upload/download client
  • Traces issue needs to be addressed
    • Merging trace information coming from pilot first
  • Timeline: Rucio side this should be done, rest is in pilot
• enum package mismatch with python 2.7 and 3.7

5 AOB