Rucio Meeting
Thursday 18 Aug 2022, 15:00
→
16:00
Europe/Zurich
Martin Barisits
(
CERN
)
Videoconference
Rucio Development Meeting
Join
Zoom Meeting ID
413496641
Host
Martin Barisits
Alternative hosts
Mario Lassnig, Cedric Serfon, Dimitrios Christidis
Passcode
28849311
Useful links
Join via phone
Zoom URL
1
News
Minutes
September meeting schedule
Summer is over!
Back to weekly meetings
Sep 1
Sep 8
Jeûne genevois
Sep 15
Sep 22
Sep 29
Component lead
update
Core & Internals
Gabriele Fronzé stepped down
Authentication & Authorisation
Rizart Dona stepped down
Deletion
Radu Carpa added
More updates to follow in the coming weeks
2
Community News & DevOps roundtable
Minutes
ATLAS
Subject DNs in Legacy format
Need to figure out how to deal with the RFC-type DNs vs. Legacy DNs
IAM-X509 exports only RFC-type DNs
CMS
1.29 upgrade
Hoped it would fix a problem with large backlog
Transfer to tape with archive feature turned on
Initial failure was that these timed out
Then Rucio submits multi-source with file already exists error
Then on next re-submission where rucio should check existing checksums/sizes it does not work either
Further try:
Set all requests to FAILED for Rucio to resubmit
Did not help either
Need to clarify if FTS is reporting the correct checksum/data
Fermilab FTS is being used, but should be the same as CERN
Radu will provide some pointers
Issue with DATASETS/CONTAINERS added to CONTAINERS do not follow the ALL grouping
Logic was never implemented
Will focus on the DATASET attached to CONTAINER case first, more complicated CONTAINER to CONTAINER one later
Belle II
Upgraded to 1.28.5 for Server and 1.28.7 for daemons
Seems to work fine
Activated temporary tables as well
Should only be the reaper for now; submitter as well if you run it on limited number of RSEs
For the rest need to go to 1.29, but need Py3 first
Need to check utilisation plots of database
DUNE
Tape backed RSEs
Use surl_algorithm for data placement
Preparing for second part of data challenge
Planned for September
Process data --> Pilots will do rucio uploads
More Rucio interactions
RAL MutliVO
Getting Authentication to work on K8s
3
Hot topics
4
Developers roundtable
Minutes
Progress-2022-08-18.png
Rucio 1.30 "The Donkeynator" progress tracking
In Progress
Introduce heartbeat handler to all daemons
#5443
[Cedric, Dimitrios, Mario]
Missing daemons: hermes, atropos, replicarecoverer
Hermes done
replicarecoverer Christoph will do it, Cedric will double check
atropos Dimitrios will check asap
Further ingetragtion of CMS-style consistency checking
#5438
[Igor]
PR for REST interface needs merging
--> 1.29.3
From Rucio core PoV things are DONE
Igor will test daemonless consistency checking then on CMS instance
Once done, next steps need to be discussed
General purpose utilities probably go to Rucio directly
Other things probably to a separate repository for others to use
Adherence to rule grouping dependent on order of ATTACH
#5251
[Igor]
Unable to Delete File DID via Undertaker
#5154
[Martin]
More complicated than initially thought; still under investigation
In Review
Done
Remove old readthedocs documentation once the new one is fully operational
#5446
[Joel, Martin]
Todo
Refactor the different test workflows to a common one [Ben, Mayank]
Create a contributor guide for probes
#84
[Dimitrios]
Create a server/daemon installation howto
#5445
[Joel]
Add symlinks for the new daemon names
#5131
[Joel]
Prepare a clear list of changes which need to be made to use new daemon names
#5132
[Joel]
Create developers testing guide in the documentation
#5452
[Joel, Mayank]
Exchange of function-based indices with normal indices
#5440
[Mario, Martin, Radu]
Document and cleanup discrepancies between surl and lfn2pfn algorithms
#5129
[Martin]
Reduce rule tickets to les than 20 [Martin]
foreign key error on deleting dids in reaper
#5733
[Martin]
Rules on containers in state OK but not all the files from the containers have locks
#5447
[Martin]
Reduce logging of server & daemons
#5449
[Mayank]
Add python 3.10 tests to CI framework.
#5145
[Mayank]
Release webUI 2.0 beta version [Mayank]
rework prometheus/statsd metric names
#5804
[Radu]
Re-work/cleanup distance table
#5454
[Radu]
rework conveyor-throttler
#5805
[Radu]
Switch to pytest [Radu?]
Needs discussion
Create documentation page for S3-type storage
#5450
[Rizart]
Exists as GGDoc (
#5284
)
Add support for S3 virtual hosted buckets
#5451
[Rizart]
Test on-premise
Ongoing support issue with AWS/Fresno X.509 shenanigans
Merge list_dids and list_dids_extended methods
#5448
[Rob]
Delayed
Other topics
5
AOB