Rucio Meeting

Martin Barisits (CERN)
Rucio Development Meeting
Zoom Meeting ID
Martin Barisits
Alternative host
Mario Lassnig
Useful links
Join via phone
Zoom URL
    • 3:00 PM 3:05 PM
      News 5m
      • Google Summer of Code 2022
        • Still a few days left!
        • Two Rucio projects
          • Rucio - S3-compatible access interface for analysis facilities
          • Rucio WebUI Revamp
      • Rucio token evolution meeting
        • Please participate in the doodle
        • Mar-03, Mar-10, Mar-17
    • 3:05 PM 3:20 PM
      Community News & DevOps roundtable 15m
      • ATLAS
        • Ongoing query optimization
        • Worker deployments
      • CMS
        • Issue with rules on containers
          • Rule does not affect all datasets
          • Attach order of dataset seems to matter
          • Martin remembers a very similar issue from some years ago
          • Stefan has a similar issue
          • Create ticket with all findings
      • Fermilab/DUNE/Rubin/...
        • Automate transfers between RSEs to generate messages and test
        • TLS change will go into 1.28
        • More complicated due to integration with OpenShift
      • Belle II
        • Moving to next DIRAC release for Py3 migration
          • v7r1 --> v7r2 --> v7r3
      • DIRAC
        • Cosmetic changes done
          • Martin can merge like this (Conveyor test fails, but this is unrelated due to flaky test)
      • ESCAPE
        • S3 endpoint integration
          • Two RSEs in ESCAPE datalake, Rizart trying to get one of them to work
          • Rucio signs URL and passes it to GFAL
          • rucio download works with Google signed URLs, but with S3 it might be a little bit different
          • --> Ticket
      • SKAO
        • Token workflow integration
          • conveyor-poller and finisher not using token workflow but falling back to X509
            • Working now
      • MultiVO/RAL
        • PR for MultiVO issue with S3 endpoints
        • S3 endpoints still not working though
          • --> Same issue as Rizart is having
    • 3:20 PM 3:50 PM
      Python 2.7 deprecation plans 30m
      • Belle II
        • Will move to v7r3 DIRAC version this summer
        • Suggested timeline should not be a problem for Belle II
        • Will move to 1.28 and stay on this even after the summer
          • Extended Support Period might be helpful, but not strictly required
      • FTS
        • Rucio does not use FTS3 clients, does direct REST interface submissions, thus FTS3 client python version does not really matter for Rucio
        • Python3 GFAL bindings
          • Need to test that specifically in Rucio
          • Seems like Python2-gfal package just seems to work fine with Python3 interpreter?
        • Python2 GFAL will still be supported for some time due to community requests
      • ESCAPE
        • Everyone seems to be on Py3 already
      • Rucio Integration Tests for TPC (dev container)
        • Still use FTS3-REST-CLI on python2
          • Python3 version is out, we should change dev container to that one
      • Conclusion
        • 1.29 python 2.7 deprecation is acceptable for all communities, we will go ahead with this plan
        • 1.28 will not be declared an Extended Support release at this point
          • Can still be done later if needed
    • 3:50 PM 3:55 PM
      Developers roundtable 5m
      • Rucio 1.28 "Teenage Mutant Ninja Donkeys" release roadmap followup
        • IN PROGRESS
          • Test framework for VO Specific tests #4797 [Mayank]
            • Flakyness fixed, some more changes to be done, then can start using it for ATLAS, other to follow
          • WebUI 2.0 Architecture #5134 [Mayank]
            • Should be done by March
          • Authentication workflows in WebUI 2.0 #5135 [Mayank]
          • Remove direct DB dependence from the WebUI #4319 [Mayank]
          • Host token evolution meeting [Martin]
            • Waiting for responses and scheduling
          • Clarify, implement and document a way how to include policy packages in containers/kubernetes #5138 [James]
            • Started to write up some guidance about policies
            • Draft PR is there - Please comment!
          • Reduce transfer tickets [Radu]
            • 5 tickets on Radu, 10 more on other people
              • 1 closed, other in progress, 
          • Disentangle fts3 specific code from conveyor poller and receiver #4873 [Radu]
          • Test SSO Login to WebUI [Rizart]
            • Keycloak SSO CERN testing
            • Also started documentation progress for others to repeat the setup
        • IN REVIEW
          • Fix broken Client and Rest API docs view #4992 [Joel]
          • Hermes 2 evolution #4610 [Cedric]
            • First PR is there, missing documentation
            • Dev container is missing Influx, Elastic, etc. would be needed for TESTING
              • Modify dev container
              • Might need some help on container modification
          • BB8 rewrite #4706 [Cedric]
            • Would be good if CMS has a look on this too
              • Eric will check if Bennedikt will look on it
            • Proceed without CMS comment for now, comments can be made later too
          • Create CMS-style consistency checking daemon #5113 [Stefcho]
            • Still responding to review comments
            • Discussion on consistency checking
          • Add MongoDB metadata plugin #5143 [Rob]
        • TODO
          • Reduce rule tickets [Martin]
          • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
          • Add symlinks for the new daemon names #5131 [Joel]
          • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
          • Clean dead code in the probes (ATLAS and common) #p/83 [Dimitrios, Eric]
            • Work required is clear, but time is the issue
            • Eric provided the list of what is not dead for CMS
          • Create a contributor guide for probes #p/84 [Dimitrios]
        • DONE
          • Create a prototype workflow with temporary tables #5130 [Radu]
          • Support prometheus metrics in daemons and servers #3416 [Radu]
          • Stabilize geoip based replica selection #5133 [Radu]
            • For refresh of the geoip database the creation time is now used
          • Write documentation page about helm-charts and versioning #h/84 [Eric]
          • Host python 2.7 deprecation meeting [Martin]
        • DELAYED
          • Include new operator engine also to json based columns #5144 [Rob]
      • Softcheck for Mypi on CI
        • Will not be final, but gets us started in type-checking our code
      • Python 3.10 tests on CI
        • Dockerfile for Centos 7 problematic
          • Integration suite uses rucio container repository
    • 3:55 PM 4:00 PM
      AOB 5m