Rucio Meeting
Thursday 19 May 2022, 15:00
→
16:00
Europe/Zurich
4/S-030 (CERN)
4/S-030
CERN
30
Show room on map
Martin Barisits
(
CERN
)
Videoconference
Rucio Development Meeting
Join
Zoom Meeting ID
413496641
Host
Martin Barisits
Alternative hosts
Mario Lassnig, Cedric Serfon, Dimitrios Christidis
Passcode
28849311
Useful links
Join via phone
Zoom URL
1
News
Minutes
June meeting schedule
June 02
June 09
June 16
CANCELLED
June 23
Next week no meeting! (Holiday @ CERN)
Ticket review
eMail to component leads out for 1:1 meetings
2
Community News & DevOps roundtable
Minutes
ATLAS
Got Amazon S3 commercial cloud to run within WLCG infrastructure
Put in Rucio documentation together with the S3 article?
Very specific documentation to Amazon
CMS
Started upgrade process to 1.28
So far on INT
Issue with DNs which have ',' inside them
Fixed by using LegacyDN
Legacy won't support unicode, special characters needs to be escaped, etc.
Added to helm-charts and containers (Option)
Getting Globus integrated
Fermilab/DUNE/Ikarus/Rubin
DUNE waiting on DataChallenge
Ikarus accessing service off-site
Everyone
Looking at argo CD (Instead of Flux)
Argo CD specifically useful for multi-cluster setups
Dennis Lee joining Rucio efforts at FNAL
DUNE
Data challenge
Staying with 1.26 LTS
Monitoring transactions and trans-atlantic throughput
QoS
Hope was to make more progress in DOMA QoS workgroup
Change of Pledging system is a barrier for established experiments
DUNE will critically rely on QoS
Need to find a way forward to make it a usable product for everyone
Token timeline in Rucio
DUNE timeline driven by RHEL goes end of service (2024)
Load into Rucio a custom proximity map
Especially for sites which do not have storage and thus no distance entry in Rucio
Belle II
Waiting on migration for 1.28
Migration to 1.29 pending then on python 3 migration of DIRAC (End of summer)
Discussion on archival features for metadata
Discussion between Rob, Cedric, Martin yesterday, Cedric will work on it for 1.29
ESCAPE
Moved from 1.28.2 to 1.28.4
Looks like after submitting a rule the submission of the transfer takes longer
Took 5h to submit a transfer
Although daemons looked not very busy
PR to fix Jupyterlab extension
Error was with the change of DIDTypes from strings to enums
SKAO
MultiVO/RAL
Issue
#5319
for implementing posix protocol with symlink functionality
Unittest would be beneficial
3
Hot topics
4
Developers roundtable
Minutes
Progress-2022-05-19.png
Rucio 1.29 "Into the donkeyverse" progress tracking
In Progress
Further ingetragtion of CMS-style consistency checking
#5438
[Igor]
Fix client API documentation page
#5444
[Joel]
Basically done, some edge cases
Remove old readthedocs documentation once the new one is fully operational
#5446
[Joel, Martin]
Ticket Cleanup Marathon [Martin, All]
Reduce logging of server & daemons
#5449
[Mayank]
Now access to ES
Started to populate dashboards about frequency distribution of log patterns
Authentication workflows in WebUI 2.0
#5135
[Mayank]
stuck on setting up base repo for webui
Add python 3.10 tests to CI framework.
#5145
[Mayank]
Fedora images are ready for 3.10
Stopped to work on CC7 - not worth it
Currently looking into centos stream 8
Setup WebUI repository for external collaboration
#w/12
[Mayank]
Update next week
Introduce heartbeat handler to all daemons
#5443
[Radu, ALL]
Started to move some daemons to the new daemon_handler
Further integration of temporary tables to other major workflows
#5441
[Radu]
Further OIDC and SSO optimization and documentation
#5437
[Rizart]
Ticket done, now looking on related ticket which will be submitted as one
Migration of SQLAlchemy queries to new query syntax (1.4 -> 2.0)
#5247
[Yuyi, ALL]
ongoing (sessions.py)
not yet too scary :-)
Radu: also work on transfer part? how to enable warnings
SetEnv PYTHONWARNINGS always::DeprecationWarning
SetEnv SQLALCHEMY_WARN_20 1
https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
Now trying to build a test to identify missing queries
In Review
Hermes 2 evolution
#4610
[Cedric]
Comments to be included
Calculation of container length and events
#5422
[Radu]
Difficult to calculate proper statistics for containers without performance degredation
Open PR, but unclear if it fully closes the issue
Done
re-evaluate the greedy = 1TB constant
#5358
[Cedric]
Clean dead code in the probes (ATLAS and common)
#83
[Eric, Dimitrios]
boto vs boto3
#5346
[James, Mario]
mypy static code analysis
#4455
[Mayank, Ben, Joel]
Discussion for type checking and documentation for developers guide coming
Flag transfer errors coming from tape sources
#5410
[Radu]
Re-factor/cleanup deletion workflow
#5442
[Radu]
Add session object to API layer
#5439
[Radu]
Remove author list from headers [Martin, Joel]
Todo
Create a contributor guide for probes
#84
[Dimitrios]
Adherence to rule grouping dependent on order of ATTACH
#5251
[Igor]
Create a server/daemon installation howto
#5445
[Joel]
Add symlinks for the new daemon names
#5131
[Joel]
Prepare a clear list of changes which need to be made to use new daemon names
#5132
[Joel]
Reduce core component tickets to less than 10 [Mario, Martin, Gabriele]
Document and cleanup discrepancies between surl and lfn2pfn algorithms
#5129
[Martin]
Unable to Delete File DID via Undertaker
#5154
[Martin]
Removal of Python 2.7 dependencies from the code
#5435
[Martin]
Reduce rule tickets to les than 10 [Martin]
Create developers testing guide in the documentation
#5452
[Mayank, Joel]
Ben working on sub-task to test locally
Remove direct DB dependence from the WebUI
#4319
[Mayank]
Analyze feedback on UI Survet
#w/13
[Mayank]
Remove full_mode switch from conveyor-receiver
#5453
[Radu]
Exchange of function-based indices with normal indices
#5440
[Radu, Mario, Martin]
Re-work/cleanup distance table
#5454
[Radu]
Create documentation page for S3-type storage
#5450
[Rizart]
Exists as GGDoc (
#5284
)
Add support for S3 virtual hosted buckets
#5451
[Rizart]
Test on-premise
Ongoing support issue with AWS/Fresno X.509 shenanigans
Update/Re-design core.meta module
#5224
[Rob]
Merge list_dids and list_dids_extended methods
#5448
[Rob]
Container linux distribution
CC7 images harder to manage (Python 3.10++)
CS8 and CS9 already in testing
Any strong preference for an OS for the official rucio images?
Can it be something else than CentOS?
ATLAS -> CentOS (Stream)
Fermilab -> CentOS
CMS offline jobs moving to alma8
Otherwise no preference for the actual rucio container
Python versions - 3.10 needed?
Not strictly needed, but would be good eventually
5
AOB
Minutes
Rucio workshop 2022
For now operating assumption, due to Covid and travel restrictions we will have the workshop at CERN in September/October
However, if any institute would like to host the workshop please get in contact with me asap