Name: Rucio Meeting
Start: 2022-06-30T15:00:00+02:00
End: 2022-06-30T16:00:00+02:00
Location: CERN

15:00 → 15:05

News 5m

July & August meeting schedule
- Bi-weekly meetings over the summer again!
- July 7
- July 21
- August 4
- August 18
- September 1 (Back to weekly schedule)
Rucio workshop 2022
- Will be co-located with the WLCG Workshop
  - Lancaster, UK
  - Nov 7-11
    - Most likely first 3 days WLCG workshop, last 2 days Rucio workshop
- Program committee still needs to be established, will start work soon
- Official announcement eMail to come soon
Rucio 1.29 "Into the Donkeyverse"
- Code-freeze since this morning!
- 1.29.0rc1 tomorrow or Monday
  - Depending on Merge progress

15:05 → 15:20

Community News & DevOps roundtable 15m

ATLAS
- RPG development
  - RPG is a system in ATLAS managing some data flows
  - Idea is to get RPG functionality into Rucio Rules & Subscriptions
  - #4508
CMS
Fermilab/Ikarus/DUNE/Rubin
- Getting DUNE policy package (lfn2pfn method) to work
- FTS3 testing for Rubin
  - Tuning transfer parameters
- Setting up cluster deployment for Rubin
DUNE
- Testing policy package for lfn2pfn method
  - Ahead of data challenge in July
  - Some issues, working on resolving these
Belle II
- Testing 1.28.5 release (Running 1.26 in production)
- Migration to 1.29 still needs DIRAC python3 deployment
ESCAPE
- Issues with replication, some transfers take too long, needs investigation
  - Might be an issue with large conveyor-submitter bulk_size
SKAO
- Transition from user_token to admin_token
  - Identified quite a few problems
    - Some Rucio related, some otherwise (FTS, ...)
    - One bug in reaper, conveyor-poller needs to be updated too
  - No group claim ('wlcg.groups') existent in token
    - Some storages might require the claim -> doesn't work
RAL/MultiVO
- K8s deployment
  - Daemons deployed
  - Server & Auth server - x509 auth does not work yet
  - proxy renewal cron needs a bit of work to renew multiple proxies for multi-vo

15:20 → 15:25

Hot topics 5m

15:25 → 15:55

Developers roundtable 30m

Rucio 1.29 "Into the donkeyverse" progress tracking
- In Progress
  - Further ingetragtion of CMS-style consistency checking #5438 [Igor]
    - Waiting for PR to be merged - then this is done (until further steps are identified)
  - Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
- In Review
  - Hermes 2 evolution #4610 [Cedric]
    - Comments to be included
  - Calculation of container length and events #5422 [Radu]
    - Difficult to calculate proper statistics for containers without performance degradation
    - Open PR, but unclear if it fully closes the issue
  - Further OIDC and SSO optimization and documentation #5437 [Rizart]
    - Ticket done, now looking on related ticket which will be submitted as one
  - Removal of Python 2.7 dependencies from the code #5435 [Martin, Joel]
  - Further integration of temporary tables to other major workflows #5441 [Radu]
- Done
  - re-evaluate the greedy = 1TB constant #5358 [Cedric]
  - Clean dead code in the probes (ATLAS and common) #83 [Eric, Dimitrios]
  - boto vs boto3 #5346 [James, Mario]
  - Fix client API documentation page #5444 [Joel]
  - Ticket Cleanup Marathon [Martin, All]
  - mypy static code analysis #4455 [Mayank, Ben, Joel]
    - Discussion for type checking and documentation for developers guide coming
  - Analyze feedback on UI Survey #w/13 [Mayank]
  - Setup WebUI repository for external collaboration #w/12 [Mayank]
  - Flag transfer errors coming from tape sources #5410 [Radu]
  - Re-factor/cleanup deletion workflow #5442 [Radu]
  - Add session object to API layer #5439 [Radu]
  - Remove author list from headers [Martin, Joel]
- Todo
  - Create a contributor guide for probes #84 [Dimitrios]
  - Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
  - Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
  - Unable to Delete File DID via Undertaker #5154 [Martin]
  - Exchange of function-based indices with normal indices #5440 [Radu, Mario, Martin]
  - Create documentation page for S3-type storage #5450 [Rizart]
    - Exists as GGDoc (#5284)
  - Add support for S3 virtual hosted buckets #5451 [Rizart]
    - Test on-premise
    - Ongoing support issue with AWS/Fresno X.509 shenanigans
  - Merge list_dids and list_dids_extended methods #5448 [Rob]
- Delayed
  - Update/Re-design core.meta module #5224 [Rob]
  - Reduce rule tickets to les than 10 [Martin]
  - Reduce core component tickets to less than 10 [Mario, Martin, Gabriele]
  - Create a server/daemon installation howto #5445 [Joel]
  - Add symlinks for the new daemon names #5131 [Joel]
    - Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
  - Migration of SQLAlchemy queries to new query syntax (1.4 -> 2.0) #5247 [Yuyi, ALL]
    - ongoing (sessions.py)
    - not yet too scary :-)
    - Radu: also work on transfer part? how to enable warnings
      - SetEnv PYTHONWARNINGS always::DeprecationWarning
        SetEnv SQLALCHEMY_WARN_20 1
        https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
    - Now trying to build a test to identify missing queries
  - Create developers testing guide in the documentation #5452 [Mayank, Joel]
    - Ben working on sub-task to test locally
  - Remove direct DB dependence from the WebUI #4319 [Mayank]
  - Reduce logging of server & daemons #5449 [Mayank]
    - Now access to ES
    - Started to populate dashboards about frequency distribution of log patterns
    - --> DELAYED for codefreeze
  - Authentication workflows in WebUI 2.0 #5135 [Mayank]
    - stuck on setting up base repo for webui
    - --> together with GSoC student, DELAYED for codefreeze
  - Add python 3.10 tests to CI framework. #5145 [Mayank]
    - Fedora images are ready for 3.10
    - Stopped to work on CC7 - not worth it
    - Currently looking into centos stream 8 and stream 9
    - --> On track
  - Remove full_mode switch from conveyor-receiver #5453 [Radu]
  - Re-work/cleanup distance table #5454 [Radu]
  - Introduce heartbeat handler to all daemons #5443 [Radu, ALL]
    - Started to move some daemons to the new daemon_handler
Other topics

15:55 → 16:00

AOB 5m

Rucio Meeting

4/S-030

CERN