Rucio Meeting

Thursday 27 Jan 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• Google Summer of Code 2022
  • If you are interested to do a Rucio project, please let me know
  • Deadline February 21
  • Two project durations: short (175h), long (350h)
• February meeting schedule
  • Feb-03 Cancelled
  • Feb-10
  • Feb-17 Python 2.7 deprecation discussion
  • Feb-24
• February release schedule
  • 1.27.4 Jan-31
  • 1.23.18 LTS Feb-07
  • 1.27.5 Feb-14
  • 1.26.11 LTS Feb-21
  • 1.27.6 Feb-28

15:05 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Database performance improvements/issues
  • http 503 incidents
    • Coming from authentication servers
    • Still under investigation
  • Network throughput test with NOTED service
    • Karlsruhe <-> TRIUMF
• CMS
  • cache-consumer almost working again
  • list_replicas on cache RSEs
    • Protocol needs to be setup
• Fermilab/DUNE/Rubin/...
  • DUNE Rucio deployment now available offsite
• Belle II
• Edinburgh/DUNE
  • Continuing with Policy packages in Containers
• RAL/MultiVO
  • S3 and IRIS IAM integration
• ESCAPE
  • Database possibly in a strange state
    • RSE usage counters vs Stats from the site
    • Script to resync the counters
    • Difference in counters could mean dark-data or dataloss
• SKAO
  • Started migrating to tokens 
    • with ESCAPE IAM

15:20 → 15:25 Hot topics

15:25 → 15:55 Developers roundtable

Progress-2022-01-27.png

• Rucio 1.28 "Teenage Mutant Ninja Donkeys" release roadmap followup
  • IN PROGRESS
    • Fix broken Client and Rest API docs view #4992 [Joel]
      • Decided on framework OpenAPI, now implementing the render
        • Will be a mixture between Py-Doc and FLASK (Flask plugin for OpenAPI)
    • Test framework for VO Specific tests #4797 [Mayank]
      • New test suite: vo-tests
      • Part left:
        • Adding configuration
        • Use composite workflows for repeated workflows (auto-test + vo-tests)
        • PR this week
    • Authentication workflows in WebUI 2.0 #5135 [Mayank]
    • WebUI 2.0 Architecture #5134 [Mayank]
    • Stabilize geoip based replica selection #5133 [Radu]
      • Issue found, test case done, PR almost finished
    • Host python 2.7 deprecation meeting [Martin]
    • Host token evolution meeting [Martin]
      • Will send out eMails this week
    • Clarify, implement and document a way how to include policy packages in containers/kubernetes #5138 [James]
      • Started experiments with policy containers
    • Include new operator engine also to json based columns #5144 [Rob]
    • Reduce transfer tickets [Radu]
      • 2 to be closed, 1 more to be opened
  • IN REVIEW
    • Hermes 2 evolution #4610 [Cedric]
      • First PR is there, missing documentation
      • Dev container is missing Influx, Elastic, etc. would be needed for TESTING
        • Modify dev container
    • BB8 rewrite #4706 [Cedric]
      • First draft already in December
      • Had discussion with CMS how they want to rebalance data
      • Query to select data is very ATLAS specific, now parameterizable
      • Would be good if CMS has a look on this too
    • Create CMS-style consistency checking daemon #5113 [Stefan]
      • PR is there, including comments from Eric
      • Update to PR probably today, no roadblocks
  • TODO
    • Reduce rule tickets [Martin]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Disentangle fts3 specific code from conveyor poller and receiver #4873 [Radu]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Clean dead code in the probes (ATLAS and common) #p/83 [Dimitrios, Eric]
    • Create a contributor guide for probes #p/84 [Dimitrios]
    • Remove direct DB dependence from the WebUI #4319 [Mayank]
    • Add MongoDB metadata plugin #5143 [Rob]
  • DONE
    • Create a prototype workflow with temporary tables #5130 [Radu]
    • Support prometheus metrics in daemons and servers #3416 [Radu]
    • Write documentation page about helm-charts and versioning #h/84 [Eric]
• Pull Request github action errors
  • Was an issue on master, fixed now
• Research on UI features to prioritise
  • WebUI would be good to get in touch with some users to get usecases for the next webUI

15:55 → 16:00 AOB