Rucio Meeting

Thursday 10 Feb 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• Next week python 2.7 deprecation discussion
• Google Summer of Code reminder
  • WebUI - Mayank
  • S3FrontEnd - Mario

15:05 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Loadbalancer updates
    • MaxConn limitation
  • Poller and Reaper optimization to reduce DB load
• CMS
• Fermilab/DUNE/Rubin/etc
• Belle II
• DUNE/Edinburgh
• DIRAC
  • MultiVO changes done
    • Cedric reviewed and approved
    • Martin will merge as a patch (most likely)
• SKAO
  • Including Token enabled storages
  • Workflow with FTS and tokens enabled
  • Found some bugs in OIDC workflow in Rucio, needs some fixes
  • Documentation
    • Some missing information
• MultiVO/STFC
  • Adding an S3 endpoint
    • S3 on the client not working?
      • get_signed_url pieces of the code working
      • Upload does not work
      • Needed to hardcode rse_id
    • James can have a look
    • Mario had a UnitTest to test, and it seemed to work
    • API/Client code asks for RSEName, Protocol, action, VO defaults to "def"

15:20 → 15:25 Hot topics

15:25 → 15:55 Developers roundtable

Progress-2022-02-10.png

• Rucio 1.28 "Teenage Mutant Ninja Donkeys" release roadmap followup
  • IN PROGRESS
    • Fix broken Client and Rest API docs view #4992 [Joel]
    • Test framework for VO Specific tests #4797 [Mayank]
      • Initial private PR is there running the VO tests for ATLAS+BelleII
      • Some flaky tests (unrelated) still need fixes
      • Then PR is ready to go
      • Go ahead with ATLAS tests, merge it, and Belle II tests can be activated later
    • Authentication workflows in WebUI 2.0 #5135 [Mayank]
    • WebUI 2.0 Architecture #5134 [Mayank]
      • Remove direct DB dependence from the WebUI #4319 [Mayank]
      • Should be done by March
    • Host python 2.7 deprecation meeting [Martin]
    • Host token evolution meeting [Martin]
      • Will send out eMails by tomorrow
    • Clarify, implement and document a way how to include policy packages in containers/kubernetes #5138 [James]
      • Started to write up some guidance about policies
      • Draft PR is there - Please comment!
    • Reduce transfer tickets [Radu]
      • 2 to be closed, 1 more to be opened
      • 5 tickets on Radu, 10 more on other people
        • Nobody answered!?
    • Disentangle fts3 specific code from conveyor poller and receiver #4873 [Radu]
  • IN REVIEW
    • Hermes 2 evolution #4610 [Cedric]
      • First PR is there, missing documentation
      • Dev container is missing Influx, Elastic, etc. would be needed for TESTING
        • Modify dev container
        • Might need some help on container modification
    • BB8 rewrite #4706 [Cedric]
      • Would be good if CMS has a look on this too
        • Eric will check if Bennedikt will look on it
    • Create CMS-style consistency checking daemon #5113 [Stefcho]
      • Still responding to review comments
      • Discussion on consistency checking
  • TODO
    • Reduce rule tickets [Martin]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Clean dead code in the probes (ATLAS and common) #p/83 [Dimitrios, Eric]
      • Work required is clear, but time is the issue
      • Eric provided the list of what is not dead for CMS
    • Create a contributor guide for probes #p/84 [Dimitrios]
    • Add MongoDB metadata plugin #5143 [Rob]
    • Test SSO Login to WebUI [Rizart]
  • DONE
    • Create a prototype workflow with temporary tables #5130 [Radu]
    • Support prometheus metrics in daemons and servers #3416 [Radu]
    • Stabilize geoip based replica selection #5133 [Radu]
      • For refresh of the geoip database the creation time is now used
    • Write documentation page about helm-charts and versioning #h/84 [Eric]
  • DELAYED
    • Include new operator engine also to json based columns #5144 [Rob]
• GFALv2 module is still in protocols
  • Remove for 1.28
• Python 3 type annotations
  • Needs some explenation how to do it
    • Part of development guide
  • Would need a higher percentage of current code to actual use it
    • People mostly copy&paste these
    • Have reviewers to remind people about it 
    • Possibly do this in multiple stages
      • Update larger parts of the code first
      • Start reminding people, but not require it yet
      • Only require later
  • mypi checking would still need to be added

15:55 → 16:00 AOB