Rucio Meeting

Thursday 17 Nov 2022, 15:00 → 16:00 Europe/Zurich

Martin Barisits (CERN)

1 News

• Thanks for participating in the 5th Rucio Workshop!
  • I hope it was informative and enjoyable for everyone
  • Videos will be available soon
  • I am still compiling an action list/followup list
• 1.30 release
  • Nov 23 8:00 Code Freeze!
    • Have your PRs submitted until then
  • RC1 on Nov 25
  • Final on Nov 30
• Next week is Thanksgiving (Nov 24/25)
  • Will still do a quick Rucio meeting, although with less attendance

2 Community News & DevOps roundtable

• ATLAS
  • 1.29.9 since today
• CMS
  • Process to upgrade to 1.29.8
  • Destination file exists error (which eventually clears)
    • Specific feature in FTS only working on tape RSEs
    • FTS change proposed
      • Also requires a change in Rucio
• Fermilab / DUNE / ICARUS / RUBIN
  • RUBIN upgraded to 1.29
    • Pulling DB secrets out of deployments
    • Getting secrets into the Rucio config file
  • Moving deployments from dev open shift cluster to prod cluster
• DUNE
  • Protocol hopping on rucio upload
    • Running on 1.26, probably fixed on later release -> 1.29
    • Possibly fixed by https://github.com/rucio/rucio/issues/4984
  • Change lfn2pfn algorithm on existing RSEs
• Belle II
• ESCAPE
  • Testing new S3 RSE
    • Upload and Download work both with x509 and tokens
    • Issue with third party copy though
      • Direct using FTS it works fine, with Rucio it does not
  • Secrets added to FTS
  • After 1.29.5 the httpd error log gives errors
  • Building 2nd Rucio cluster
    • Using terraform to initialize clusters
    • Will send ATLAS terraform script

3 Hot topics

• Turning camera on for Rucio meeting
  • Was requested by some participants of the Rucio workshop
  • CERN has somewhat of a culture of having our cameras off and do audio-only meetings
  • I find turning the camera on in principle a good idea
    • More focus, increases engagement, sense of community, ...
    • Not mandatory, of course
• Slack persistent chat history
  • Move to CERN Mattermost?
  • Registration of CERN lightweight account super easy
    • Dennis tried it, was very fast

4 Developers roundtable

Progress-2022-11-17.png

• Rucio 1.30 "The Donkeynator" progress tracking
  • Release date will be end of November
    • Code freeze on Nov-23
  • In Progress​​
    • Create a contributor guide for probes #84 [Dimitrios]
    • Exchange `Tombstone` function-based indices with normal indices #5440 [Mario, Martin, Radu]
      • Will only include one index (Replicas tombstone idx) for 1.30
    • Unable to Delete File DID via Undertaker #5154 [Martin]
      • Does not work on 1.26
      • Need unit test on 1.29
    • Rules on containers in state OK but not all the files from the containers have locks #5447 [Martin]
    • Create developers testing guide in the documentation #5452 [Joel, Mayank]
    • Add python 3.10 tests to CI framework. #5145 [Mayank]
      • Tests with sqllite fail to bootstrap on stream8/9 because of timeout
    • Release webUI 2.0 beta version [Mayank]
      • container is in place
      • helm-charts testing right now
      • Followup on authentication verification for x509 and oidc tokens
      • After beta release, polish the user pages
    • Switch to pytest [Radu]
      • Will be finished by end of the week
    • Re-work/cleanup distance table #5454 [Radu]
      • Will be done
  • In Review
    • rework prometheus/statsd metric names #5804 [Radu]
  • Done
    • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
    • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
    • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
    • Introduce heartbeat handler to all daemons #5443 [Cedric]
    • rework conveyor-throttler #5805 [Radu]
    • Create documentation page for S3-type storage #5450 [Rizart]
  • Todo
    • Create a server/daemon installation howto #5445 [Joel, Mayank]
      • To discuss with Martin, Mayank, Anton
    • Merge list_dids and list_dids_extended methods #5448 [Rob]
  • Delayed
    • Refactor the different test workflows to a common one [Ben, Mayank]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Reduce rule tickets to les than 20 [Martin]
    • Reduce logging of server & daemons #5449 [Mayank
    • Add support for S3 virtual hosted buckets #5451 [Rizart]
      • Test on-premise
      • Ongoing support issue with AWS/Fresno X.509 shenanigans
    • foreign key error on deleting dids in reaper #5733 [Martin]
• Other topics
  •  

5 AOB