Rucio Meeting

Thursday 17 Feb 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

1 News

• Google Summer of Code 2022
  • Still a few days left!
  • Two Rucio projects
    • Rucio - S3-compatible access interface for analysis facilities
    • Rucio WebUI Revamp
• Rucio token evolution meeting
  • Please participate in the doodle
  • Mar-03, Mar-10, Mar-17

2 Community News & DevOps roundtable

• ATLAS
  • Ongoing query optimization
  • Worker deployments
• CMS
  • Issue with rules on containers
    • Rule does not affect all datasets
    • Attach order of dataset seems to matter
    • Martin remembers a very similar issue from some years ago
    • Stefan has a similar issue
    • Create ticket with all findings
• Fermilab/DUNE/Rubin/...
  • Automate transfers between RSEs to generate messages and test
  • TLS change will go into 1.28
  • More complicated due to integration with OpenShift
• Belle II
  • Moving to next DIRAC release for Py3 migration
    • v7r1 --> v7r2 --> v7r3
• DIRAC
  • Cosmetic changes done
    • Martin can merge like this (Conveyor test fails, but this is unrelated due to flaky test)
• ESCAPE
  • S3 endpoint integration
    • Two RSEs in ESCAPE datalake, Rizart trying to get one of them to work
    • Rucio signs URL and passes it to GFAL
    • rucio download works with Google signed URLs, but with S3 it might be a little bit different
    • --> Ticket
• SKAO
  • Token workflow integration
    • conveyor-poller and finisher not using token workflow but falling back to X509
      • Working now
• MultiVO/RAL
  • PR for MultiVO issue with S3 endpoints
  • S3 endpoints still not working though
    • --> Same issue as Rizart is having

3 Python 2.7 deprecation plans

• Belle II
  • Will move to v7r3 DIRAC version this summer
  • Suggested timeline should not be a problem for Belle II
  • Will move to 1.28 and stay on this even after the summer
    • Extended Support Period might be helpful, but not strictly required
• FTS
  • Rucio does not use FTS3 clients, does direct REST interface submissions, thus FTS3 client python version does not really matter for Rucio
  • Python3 GFAL bindings
    • Need to test that specifically in Rucio
    • Seems like Python2-gfal package just seems to work fine with Python3 interpreter?
  • Python2 GFAL will still be supported for some time due to community requests
• ESCAPE
  • Everyone seems to be on Py3 already
• Rucio Integration Tests for TPC (dev container)
  • Still use FTS3-REST-CLI on python2
    • Python3 version is out, we should change dev container to that one
• Conclusion
  • 1.29 python 2.7 deprecation is acceptable for all communities, we will go ahead with this plan
  • 1.28 will not be declared an Extended Support release at this point
    • Can still be done later if needed

4 Developers roundtable

Progress-2022-02-17.png

• Rucio 1.28 "Teenage Mutant Ninja Donkeys" release roadmap followup
  • IN PROGRESS
    • Test framework for VO Specific tests #4797 [Mayank]
      • Flakyness fixed, some more changes to be done, then can start using it for ATLAS, other to follow
    • WebUI 2.0 Architecture #5134 [Mayank]
      • Should be done by March
    • Authentication workflows in WebUI 2.0 #5135 [Mayank]
    • Remove direct DB dependence from the WebUI #4319 [Mayank]
    • Host token evolution meeting [Martin]
      • Waiting for responses and scheduling
    • Clarify, implement and document a way how to include policy packages in containers/kubernetes #5138 [James]
      • Started to write up some guidance about policies
      • Draft PR is there - Please comment!
    • Reduce transfer tickets [Radu]
      • 5 tickets on Radu, 10 more on other people
        • 1 closed, other in progress, 
    • Disentangle fts3 specific code from conveyor poller and receiver #4873 [Radu]
    • Test SSO Login to WebUI [Rizart]
      • Keycloak SSO CERN testing
      • Also started documentation progress for others to repeat the setup
  • IN REVIEW
    • Fix broken Client and Rest API docs view #4992 [Joel]
    • Hermes 2 evolution #4610 [Cedric]
      • First PR is there, missing documentation
      • Dev container is missing Influx, Elastic, etc. would be needed for TESTING
        • Modify dev container
        • Might need some help on container modification
    • BB8 rewrite #4706 [Cedric]
      • Would be good if CMS has a look on this too
        • Eric will check if Bennedikt will look on it
      • Proceed without CMS comment for now, comments can be made later too
    • Create CMS-style consistency checking daemon #5113 [Stefcho]
      • Still responding to review comments
      • Discussion on consistency checking
    • Add MongoDB metadata plugin #5143 [Rob]
  • TODO
    • Reduce rule tickets [Martin]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Clean dead code in the probes (ATLAS and common) #p/83 [Dimitrios, Eric]
      • Work required is clear, but time is the issue
      • Eric provided the list of what is not dead for CMS
    • Create a contributor guide for probes #p/84 [Dimitrios]
  • DONE
    • Create a prototype workflow with temporary tables #5130 [Radu]
    • Support prometheus metrics in daemons and servers #3416 [Radu]
    • Stabilize geoip based replica selection #5133 [Radu]
      • For refresh of the geoip database the creation time is now used
    • Write documentation page about helm-charts and versioning #h/84 [Eric]
    • Host python 2.7 deprecation meeting [Martin]
  • DELAYED
    • Include new operator engine also to json based columns #5144 [Rob]
• Softcheck for Mypi on CI
  • Will not be final, but gets us started in type-checking our code
• Python 3.10 tests on CI
  • Dockerfile for Centos 7 problematic
    • Integration suite uses rucio container repository

5 AOB