Rucio Meeting

Thursday 10 Mar 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

1 News

• 1.28 outlook
  • Code Freeze next Wednesday!
  • 1.28.0rc1 Mar-18
  • 1.28.0 Mar-25
• Token discussion conclusion
  • Martin preparing a documentation page with the rucio token plans

2 Community News & DevOps roundtable

• ATLAS
  • Subscriptions for the tape challenge next week
    • Right now only inclusive filter
      • Cedric: You can exclude patterns, but not metadata
    • Cedric: Might be able to be done via exclusion pattern, but can also make this more flexible
    • What does split_rule mean?
      • rse_expression (multiple rses) -> get one rse
  • CERN cloud issue
    • Maintanence cleanup script on k8s clusters had a typo
      • -> Script was wiping active clusters, including 3 out 4 atlas clusters
    • Documentation and testing of cluster creation scenario is key
  • CERN Summer student
• CMS
  • CMS also hit by the CERN cloud issue
    • Eric unfortunately on vacation, Nick was able to recover the cluster
  • Next week deletion campaign
    • Monitor the DB and reapers
  • list collection replica difference between normal + --deep
    • Under investigation
• Fermilab, DUNE, Icarus, etc.
  • DUNE issues with transfers
    • Possibly fts3 based
  • RUBIN
    • Transferring files between three continents now!
    • Defining the data structure and metadata catalog
    • Will be tested NCSA -> SLAC transfers 
    • K8s virtual clusters
      • Migrating rubin from docker based rucio to k8s based rucio
• Belle II
  • Migration to python3 ongoing
    • Expected to be done in June
• DUNE/Edinburgh
  • More tests on James DUNE Rucio work soon
• MultiVO/RAL
  • Move to test-k8s instance
  • S3 endpoints
    • Use `strict_copy` rse attribute
    • Several authentication errors at the moment
• SKAO

3 Hot topics

4 Developers roundtable

Progress-2022-03-10.png

• Rucio 1.28 "Teenage Mutant Ninja Donkeys" release roadmap followup
  • IN PROGRESS
    • WebUI 2.0 Architecture #5134 [Mayank]
      • Base repository now in place
      • github.com/rucio/webui
      • Some document this/next week focusing on authentication workflows
    • Authentication workflows in WebUI 2.0 #5135 [Mayank]
    • Remove direct DB dependence from the WebUI #4319 [Mayank]
    • Test SSO Login to WebUI [Rizart]
      • Very close to a running prototype now
      • CERN SSO cannot really define custom scopes which might affect us in the future
        • audience claims should be there
  • IN REVIEW
    • Hermes 2 evolution #4610 [Cedric]
      • First PR is there, missing documentation
      • Dev container is missing Influx, Elastic, etc. would be needed for TESTING
        • Modify dev container
        • Might need some help on container modification
    • BB8 rewrite #4706 [Cedric]
      • Would be good if CMS has a look on this too
        • Eric will check if Bennedikt will look on it
      • Proceed without CMS comment for now, comments can be made later too
    • Test framework for VO Specific tests #4797 [Mayank]
      • Minor comments to include, good to merge otherwise
    • Add MongoDB metadata plugin #5143 [Rob]
      • Waiting on the mongodb dependency PR
        • This needs to be merged first, for #5143 to go forward
      • Waiting on review - but should be all done 
        • Relies on a second PR for testing (mongodb dependency)
    • Include new operator engine also to json based columns #5144 [Rob]
  • TODO
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Clean dead code in the probes (ATLAS and common) #p/83 [Dimitrios, Eric]
      • Work required is clear, but time is the issue
      • Eric provided the list of what is not dead for CMS
    • Create a contributor guide for probes #p/84 [Dimitrios]
  • DONE
    • Create a prototype workflow with temporary tables #5130 [Radu]
    • Support prometheus metrics in daemons and servers #3416 [Radu]
    • Stabilize geoip based replica selection #5133 [Radu]
      • For refresh of the geoip database the creation time is now used
    • Reduce transfer tickets [Radu]
    • Disentangle fts3 specific code from conveyor poller and receiver #4873 [Radu]
    • Write documentation page about helm-charts and versioning #h/84 [Eric]
    • Fix broken Client and Rest API docs view #4992 [Joel]
      • REST API online on rucio.cern.ch/documentation
      • 60% of the endpoints are already there
    • Host python 2.7 deprecation meeting [Martin]
    • Host token evolution meeting [Martin]
    • Create CMS-style consistency checking daemon #5113 [Stefan]
    • Clarify, implement and document a way how to include policy packages in containers/kubernetes #5138 [James]
  • DELAYED
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Reduce rule tickets [Martin]
• Consistency checks w/ AGIS (Fabio)
  • Review todo list for Dimitrios
• auditor bz2file dependency replacement
  • Needs some comments!

5 AOB