Rucio Meeting

Thursday 2 Mar 2023, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

15:00 → 15:05 News

• March release schedule
  • 1.30.7 Mar-13
  • 1.31 "Donkeys of the Caribbean"
    • Code Freeze on Mar-15
    • 1.31.0rc1 Mar-17
    • 1.31 final Mar-24

15:05 → 15:25 Community News & DevOps roundtable

• ATLAS
  • New tape REST API testing
    • Rucio -> FTS -> Storage
      • Works with CTA
      • Now testing dCache
  • ALMA9 migration
    • Move of ATLAS auth server to k8s (Which comes also with move to alma9)
    • We are now running everything on ALMA9, except the reaper
      • Problem with reaper is the usage of openssl 3 with ALMA9
        • CPU consumption increased by a factor 10
        • Known issue in OpenSSL connected to the loading of certificates (lots in our case)
          • Developers position: Will not change
          • Two changes needed
            • GFAL needs to change certificate loading (load all certificates once, re-use object later) #TICKET
              • Right now each request re-loads all certificates
            • Rucio might need a change in using the gfal context as well #TICKET
        • OpenSSL 3.1 not tried yet
          • Not a way forward anyway, more a question of curiosity
  • Test instead of Rucio + FTS for data movement to NERSC to use Rucio + Globus
    • HPC folks now ready
• CMS
  • 1.30 upgrade was done
  • Probes are still pinned to 1.29
  • Helm chart upgrade for auth servers made a problem, under investigation
  • Old webUI not working, still investigating
  • Testing new WebUI with oidc-auth
    • Probably best to wait a bit more, due to the re-write Mayank is currently doing
  • Kronos issue at the moment
    • Increased memory, which prolonged interval between crashes
    • Disconnect from activemq issue
    • Don't see this in ATLAS
    • Try to move to newer STOMP version (8.0.1, current) 
    • Kubernetes cluster issue with maximum connection?
• Fermilab/DUNE/RUBIN/ICARUS
  • Debugging transfer issues for Icarus
    • Site did not update voms lib
  • RUBIN debugging ongoing for selection of checksum algorithm
    • SLAC --> CC-IN2P3
  • DUNE
    • Plan for updating path column for all replicas on castor@CERN
      • Possibly also needs to be done for dCache
      • ~1.2M records in replicas
    • Bugfixes for token-renewal
      • After update for 1.29.11 renewal works, but webui breaks now #6099
• DUNE
  • Protocols re-order discussion from two weeks ago: Would be good to get this into LTS release
  • gfal not communicating create parent_directory (if not exists)
    • Dimitrios: Make sure the scheme is davs, instead of https
    • Doug: Did not seem to make any difference, but will try again
• Belle II
  • Metadata evolution in B2
    • Move to Rucio Metadata in the coming months
    • did_meta ~60M rows
• Intertwin
  • Dijana introduction
  • Currently setting up Intertwin Testbed @ DESY
    • OIDC via EGI logon (Needed some tweaks due to no scopes in EGI logon)
• Edinburgh / DUNE
  • Working on Metacat plugin for Rucio

15:25 → 15:55 Developers roundtable

Progress-2023-03-02.png

• 1.31 "Donkeys of the Caribbean" Priority followup
  • In Progress
    • Create ongoing token architecture document [Dimitrios]
    • Collect feedback from running ATLAS webui beta [Mayank]
    • Finalize authentication workflows for webUI #w/125 [Mayank]
    • Merge list_dids and list_dids_extended methods #5448 [Rob]
    • Track SQLAlchemy 2.0 migration progress [Yuyi, Martin]
      • Ticket updated with an overview
    • Replace hermes1 by hermes2 #6089 [Mario, Cedric]
      • Patch is ready (pyright!)
  • In Review
    • Rules on containers in state OK but not all the files from the containers have locks #5447 [Anton, Martin]
    • Unable to Delete File DID via Undertaker #5154 [Martin, Anton]
  • Todo
    • Exchange of tombstone function-based indices with normal indices #5440 [Martin]
    • Create developers testing guide in the documentation #5452 [Mayank, Radu]
      • Radu can work on IntelliJ setup part
      • Mayank working on the vscode part
      • Setup will go into dev containers
        • BEst would be to have a generic one which works for both IDEs
  • Done
    • Add python 3.10 tests to CI framework. #5145 [Mayank]
    • STOMP connections not closed by Hermes? #5894 [Eric, Yuyi]
    • Reduce rule tickets to <13 [Martin, Anton]
    • WebUI release process #121 [Mayank]
    • Move WebUI Core to Clean + Hexagonal Architecture + Domain Driven Design #117 [Mayank]
    • Increase WebUI Test coverage [Mayank]
      • 2/4 Test suites are in place;
  • Delayed
    • Create a server/daemon installation howto #5445 [Mayank]
    • foreign key error on deleting dids in reaper #5733 [Martin, Cedric]
• Documentation corner
  • Ambiguity of "developing" and "developer" #25 [Eraldo]
  • Document environmental variables affecting the client #171 [Dimitrios]
  • Documentation and dev guidelines for MyPi type annotations #116 [Mayank, Martin]
  • Create developers testing guide in the documentation #177 [Mayank] [1.31 priority]
  • Create a server/daemon installation howto #178 [Mayank]
• Other topics
  •  

15:55 → 16:00 AOB