Rucio Meeting

Thursday Aug 18, 2022, 3:00 PM → 4:00 PM Europe/Zurich

Martin Barisits (CERN)

1 News

• September meeting schedule
  • Summer is over! 
    • Back to weekly meetings
  • Sep 1
  • Sep 8 Jeûne genevois
  • Sep 15
  • Sep 22
  • Sep 29
• Component lead update
  • Core & Internals
    • Gabriele Fronzé stepped down
  • Authentication & Authorisation
    • Rizart Dona stepped down
  • Deletion
    • Radu Carpa added
  • More updates to follow in the coming weeks

2 Community News & DevOps roundtable

• ATLAS
  • Subject DNs in Legacy format
    • Need to figure out how to deal with the RFC-type DNs vs. Legacy DNs
    • IAM-X509 exports only RFC-type DNs
• CMS
  • 1.29 upgrade
    • Hoped it would fix a problem with large backlog
      • Transfer to tape with archive feature turned on
        • Initial failure was that these timed out
      • Then Rucio submits multi-source with file already exists error
      • Then on next re-submission where rucio should check existing checksums/sizes it does not work either
    • Further try:
      • Set all requests to FAILED for Rucio to resubmit
        • Did not help either
    • Need to clarify if FTS is reporting the correct checksum/data
      • Fermilab FTS is being used, but should be the same as CERN
      • Radu will provide some pointers
  • Issue with DATASETS/CONTAINERS added to CONTAINERS do not follow the ALL grouping
    • Logic was never implemented
    • Will focus on the DATASET attached to CONTAINER case first, more complicated CONTAINER to CONTAINER one later
• Belle II
  • Upgraded to 1.28.5 for Server and 1.28.7 for daemons
    • Seems to work fine
  • Activated temporary tables as well
    • Should only be the reaper for now; submitter as well if you run it on limited number of RSEs
    • For the rest need to go to 1.29, but need Py3 first
    • Need to check utilisation plots of database
• DUNE
  • Tape backed RSEs
    • Use surl_algorithm for data placement
  • Preparing for second part of data challenge
    • Planned for September
    • Process data --> Pilots will do rucio uploads
    • More Rucio interactions
• RAL MutliVO
  • Getting Authentication to work on K8s

3 Hot topics

4 Developers roundtable

Progress-2022-08-18.png

• Rucio 1.30 "The Donkeynator" progress tracking
  • In Progress​​
    • Introduce heartbeat handler to all daemons #5443 [Cedric, Dimitrios, Mario]
      • Missing daemons: hermes, atropos, replicarecoverer
        • Hermes done
        • replicarecoverer Christoph will do it, Cedric will double check
        • atropos Dimitrios will check asap
    • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
      • PR for REST interface needs merging
        • --> 1.29.3
      • From Rucio core PoV things are DONE
      • Igor will test daemonless consistency checking then on CMS instance
        • Once done, next steps need to be discussed
          • General purpose utilities probably go to Rucio directly
          • Other things probably to a separate repository for others to use
    • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
    • Unable to Delete File DID via Undertaker #5154 [Martin]
      • More complicated than initially thought; still under investigation
  • In Review
  • Done
    • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
  • Todo
    • Refactor the different test workflows to a common one [Ben, Mayank]
    • Create a contributor guide for probes #84 [Dimitrios]
    • Create a server/daemon installation howto #5445 [Joel]
    • Add symlinks for the new daemon names #5131 [Joel]
    • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Create developers testing guide in the documentation #5452 [Joel, Mayank]
    • Exchange of function-based indices with normal indices #5440 [Mario, Martin, Radu]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Reduce rule tickets to les than 20 [Martin]
    • foreign key error on deleting dids in reaper #5733 [Martin]
    • Rules on containers in state OK but not all the files from the containers have locks #5447 [Martin]
    • Reduce logging of server & daemons #5449 [Mayank]
    • Add python 3.10 tests to CI framework. #5145 [Mayank]
    • Release webUI 2.0 beta version [Mayank]
    • rework prometheus/statsd metric names #5804 [Radu]
    • Re-work/cleanup distance table #5454 [Radu]
    • rework conveyor-throttler #5805 [Radu]
    • Switch to pytest [Radu?]
      • Needs discussion
    • Create documentation page for S3-type storage #5450 [Rizart]
      • Exists as GGDoc (#5284)
    • Add support for S3 virtual hosted buckets #5451 [Rizart]
      • Test on-premise
      • Ongoing support issue with AWS/Fresno X.509 shenanigans
    • Merge list_dids and list_dids_extended methods #5448 [Rob]
  • Delayed
• Other topics

5 AOB