Rucio Development Meeting

Thursday 12 Dec 2019, 15:00 → 16:30 Europe/Zurich

4/S-056 (CERN)

Martin Barisits (CERN)

15:00 → 15:10 News

• Release schedule
  • 1.21.3 release yesterday
  • Next release 1.21.4 early January
• Next Development meeting
  • January 9th 
  • New room for next year!
    • 4/S-030
• New CERN technical student for Rucio next year

15:10 → 15:30 News from the experiments

• Belle II
  • Discussion how to partition the namespace
  • Testing tool for migration
    • Should be done in January
    • Schedule a talk in January in Dev Meeting
  • LFC has hierachical structure of namespace
    • Might be necessary to replicate in Rucio with levels of Containers
• CMS
  • No news
• Ian MultiVO/STFC Communities
  • No news
• James DUNE/Others
  • Make a package for DUNE
  • Will try one for ATLAS as well (Martin)
  • Needs to be integrated with MultiVO as well
  • Need to see how we handle policy package in the containers
• ATLAS
  • A few headaches with overload of the database
  • Found inconsistencies in the DB (lock_cnt without locks etc)
    • Unclear if historical
    • In this case it prevents deletion

15:30 → 15:50 Hot topics

• Client upload seems to be slow? (See eMail from Asoka)
  • Unclear yet, trying to reproduce;

15:50 → 16:20 Developers roundtable

Rucio_Issue_burn_2019-12-12.png

Rucio_priority_progress_2019-12-12.png

• Issue burn update
• Rucio 1.22 "Green Donkey" priority development followup
  • In Progress
    • Kubernetes [Thomas]
      • pre-GDB talk this week
      • Logging does still not work properly
      • Tested the Auto-Scaling a bit
      • Some daemons take a lot of memory, current bug in kubernetes which crashes the cluster in case of limit hits
      • Network driver might be cause of slow HTTPD servers (due to communication with database, thus this should affect everything)
      • Traces: Dedicated pool of servers for traces?
    • Reaper 2.0 #2412 [Cedric, Thomas]
      • It's essentially done, a few things need to be improved
      • Query to get list of unlocked replicas needs to be improved
      • Protection has been relaxed, but need to check if it is too much (needs review)
      • Exclude RSEs with timeout: Not yet
      • Archive part: Mario or Martin can maybe help
      • Reaper1 vs Reaper2 deletion of nonexistent file is not reported
    • OIDC #2612 [Jaroslav]
    • MultiVO #2635 [Ian, Andrew]
      • Martin will merge this, this or next week
      • Further development when Eli starts
    • Repository cleanup #5252 [Martin, Thomas]
      • Separate tools to a separate repo 
      • Maybe merge with probes repo
    • RSEMgr v2.0 #3147 [Tomas, Tobi]
      • Upload function moved to upload client
        • Tests work
      • Rebase will fix conflict
      • We should do upload/download testing with RSEs in Travis
        • Check with [Mario] to get the docker development also into travis
    • Unification of metadata #3096 [Aris]
      • Most of the work should be done by next week
    • General update of BB8 #3148 [Tomas, JJ, Dimitrios]
      • PR submitted by JJ, needs lots of review
    • Cleanup of Issues [Everyone]
      • Update tickets
  • ToDo
    • Data Pre-Placement [Thomas]
    • Documentation [Martin]
    • rucio.cfg vs config table #2630 [Mario]
    • Versioned history tables need explicit definition #2063 [Martin]
    • Protection of sources too strict in reaper #1637 [Cedric, Thomas]
    • Add file to content history in reaper #37 [Cedric, Thomas]
    • Reaper should stop in case of judge backlog #1578 [Martin]
    • Handling of Archives in Reaper #1431 [Cedric, Thomas, Mario, Martin]
    • Missing features for Archives #1091 [Mario, Martin, Tobi, Tomas]
    • Documentation page listing config table and RSE attributes #2631 [Mario]
    • Operators doc and repo #2636 [Martin]
  • Done
    • Configuration / Permissions / Policy #533 [James]
      • Waiting for comments on latest PR

16:20 → 16:30 AOB