Rucio Meeting

Thursday 5 May 2022, 15:00 → 16:00 Europe/Zurich

4/S-030 (CERN)

Martin Barisits (CERN)

15:00 → 15:05 News

• Component lead update
  • Fabio replacing Dimitrios in Consistency checks
    • Component leads now Fabio, Igor
• Next week
  • Talk by Franck Pachot on YugabyteDB
• May Release schedule
  • 1.28.3 May-02
  • 1.28.4 May-16
  • 1.28.5 May-30
  • 1.26.13 LTS May-30
  • 1.23.20 LTS May-30
• May meeting schedule
  • May-05
  • May-12
  • May-19
  • May-26 CANCELLED (Holiday at CERN)
• Ticket review marathon

15:05 → 15:20 Community News & DevOps roundtable

• ATLAS
  • Deployment related topics
  • Open point on heavy necromancer sql query
  • Database hickup yesterday
    • K8s deployment "re-started" on its own
• CMS
  • Same database issue as ATLAS?
  • Blocker for upgrading to 1.28 is gone
    • Upgrade in the next 1-2 weeks
  • DNs with commas
    • Testing a patch which should fix the issue
• Fermilab, DUNE, Rubin, ...
  • Need to test new policy package for DUNE (lfn2pfn algorithms etc.)
    • Data challenge scheduled for mid-June
  • Rucio instances
    • Move from bash scripts to flux for manage configuration
      • Flux only compatible with OKD4
  • RUBIN
    • Need to move all data to SLAC
      • Using rsync (Preserve permissions etc.)
      • Possibly to use Rucio rclone protocol to do it?
    • Will point to presentation for rclone in Rucio
• Belle II
  • Working on Belle II specific VO tests
    • Some other tests still failing due to naming conventions
• Edinburgh/DUNE
  • Policy package
    • Add metadata plugins to policy package?
      • To discuss
    • Inclusion of session in policies 
      • This needs to be announced to everyone so they can include it in their policy package
      • Make use of versioning of policy packages as well?
• ESCAPE
  • 1.26 -> 1.28 upgrade is done!
  • jupyterlab extension
    • Martin will push a new package

15:20 → 15:25 Hot topics

15:25 → 15:55 Developers roundtable

Progress-2022-05-05.png

• Rucio 1.29 "Into the donkeyverse" progress tracking
  • In Progress​​
    • Hermes 2 evolution #4610 [Cedric]
      • Added support for influx, elastic and activeMQ to the container
      • Mayank will help
    • Further ingetragtion of CMS-style consistency checking #5438 [Igor]
    • Fix client API documentation page #5444 [Joel]
    • Reduce logging of server & daemons #5449 [Mayank]
      • script to map raw logs, waiting for MONIT
      • Still waiting for access to ES
    • Authentication workflows in WebUI 2.0 #5135 [Mayank]
      • stuck on setting up base repo for webui
    • Add python 3.10 tests to CI framework. #5145 [Mayank]
      • ongoing, by next week
      • Difficult to get CC7 to work with py3.10
    • Setup WebUI repository for external collaboration #w/12 [Mayank]
    • Calculation of container length and events #5422 [Radu]
      • Difficult to calculate proper statistics for containers without performance degredation
    • Introduce heartbeat handler to all daemons #5443 [Radu, ALL]
      • Started to move some daemons to the new daemon_handler
    • Further OIDC and SSO optimization and documentation #5437 [Rizart]
    • Migration of SQLAlchemy queries to new query syntax (1.4 -> 2.0) #5247 [Yuyi, ALL]
      • ongoing (sessions.py)
      • not yet too scary :-)
      • Radu: also work on transfer part? how to enable warnings
        • SetEnv PYTHONWARNINGS always::DeprecationWarning
          SetEnv SQLALCHEMY_WARN_20 1
          https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
  • In Review
    • boto vs boto3 #5346 [James, Mario]
    • mypy static code analysis #4455 [Mayank, Ben, Joel]
      • New workflow will test the annotations of new PRs (Amount of un-annotated code should not increase)
      • PR does not type-check yet, since too many type-annotations are missing
    • Re-factor/cleanup deletion workflow #5442 [Radu]
    • Add session object to API layer #5439 [Radu]
  • Done
    • re-evaluate the greedy = 1TB constant #5358 [Cedric]
    • Clean dead code in the probes (ATLAS and common) #83 [Eric, Dimitrios]
    • Flag transfer errors coming from tape sources #5410 [Radu]
    • Remove author list from headers [Martin, Joel]
  • Todo
    • Create a contributor guide for probes #84 [Dimitrios]
    • Adherence to rule grouping dependent on order of ATTACH #5251 [Igor]
    • Create a server/daemon installation howto #5445 [Joel]
    • Remove old readthedocs documentation once the new one is fully operational #5446 [Joel, Martin]
    • Add symlinks for the new daemon names #5131 [Joel]
      • Prepare a clear list of changes which need to be made to use new daemon names #5132 [Joel]
    • Reduce core component tickets to less than 10 [Mario, Martin, Gabriele]
    • Document and cleanup discrepancies between surl and lfn2pfn algorithms #5129 [Martin]
    • Unable to Delete File DID via Undertaker #5154 [Martin]
    • Removal of Python 2.7 dependencies from the code #5435 [Martin]
    • Ticket Cleanup Marathon [Martin, All]
    • Reduce rule tickets to les than 10 [Martin]
    • Create developers testing guide in the documentation #5452 [Mayank, Joel]
      • Ben working on sub-task to test locally
    • Remove direct DB dependence from the WebUI #4319 [Mayank]
    • Analyze feedback on UI Survet #w/13 [Mayank]
    • Further integration of temporary tables to other major workflows #5441 [Radu]
    • Remove full_mode switch from conveyor-receiver #5453 [Radu]
    • Exchange of function-based indices with normal indices #5440 [Radu, Mario, Martin]
    • Re-work/cleanup distance table #5454 [Radu]
    • Create documentation page for S3-type storage #5450 [Rizart]
      • Exists as GGDoc (#5284)
    • Add support for S3 virtual hosted buckets #5451 [Rizart]
      • Test on-premise
      • Ongoing support issue with AWS/Fresno X.509 shenanigans
    • Update/Re-design core.meta module #5224 [Rob]
    • Merge list_dids and list_dids_extended methods #5448 [Rob]

15:55 → 16:00 AOB

• Hackathon
  • Maybe a hackathon before the summer? Any interest?
• Amazon certificate change
  • Cannot use non-amazon/custom certificates