Rucio Development Meeting
Thursday 31 Oct 2019, 15:00
→
16:30
Europe/Zurich
4/S-056 (CERN)
4/S-056
CERN
20
Show room on map
Martin Barisits
(
CERN
)
1
News
Minutes
Rucio 1.21.0 "Donkeys of the Galaxy" Code Freeze on Sunday 23:59 CEST
Every PR not submitted by then will not be included in the feature release
2
News from the experiments
Minutes
ESCAPE
Escape CRIC instance with real escape data
Start probes via nagios or not?
Probably not needed to just run probe
Change rse_expression_parser test to initialize RSEs in the setup
ATLAS
-
CMS
Generalize probes (schema) so it is usable by other communities
Bring probes to K8s
Thomas: Some probes can be run as crons, others we should maybe change to report stats to prometheus
CMS has framework in place to send stuff to prometheus already
Probes which report table sizes (messages, etc.) to statsd (graphite) could just send it to prometheus instead
Possibly should also change the internal measurements to go to prometheus instead of graphite
Traces & Popularity system
Meeting after CHEP about this
Multi-VO
Move Andrews open PR to next feature release
3
Hot topics
a) 2020 Release schedule
Minutes
Speaker
:
Martin Barisits
(
CERN
)
2020 release schedule.pdf
Agreed
4
Developers roundtable
Minutes
Rucio 1.21 priority followup
Focus
Cleanup & Stability
Documentation
Deployment (Kubernetes!)
Open ID connect
#2612
Provisioning client used similar to query VOMS (--> probes)
Will need a release candidate to test with e.g. ESCAPE
Rucio.cfg vs Rucio config table cleanup
#2630
Will start a document and have every component responsible to comment;
Move to next one
Documentation for configuration parameters
#2631
History table definition explicitly (No Versioned models)
#2063
PR prepared
MultiVO features
#2635
Open PR from Andrew, moved to next feature
Reaper 2.0
#2412
Needs improvement in the query to get list of unlocked replicas
SKIP LOCKED does not work due to being used as a subquery
Needs some more work with the query
Source protection: Implemented
Some minor other things might need improvement too
ATLAS running all Tier-3s, selected Tier-2s and one Tier-1 (RAL)
60 threads (partly on K8s)
Operators documentation / recipes
#2636
Expand Kubernetes usage
ATLAS: By the end of the month run all daemons/server on integration cluster
DONE
Next year: Move all production services to production as well
Depends on experience with integration service now
K8s receives 1-2% of the load now
Performance of http servers still a question
Increase load (5-10%) on ATLAS cluster in K8s
Using CERN IT infrastructure to collect logs to Hadoop and ES
Tracking what happened with a did
#2637
Hannes submitted a PR, needs review
XCache config table population add to probe
#2638
Needs to be commited
BB8 Needs better configuration and get rid of hard-coded entries
Better way to deal with configuration/permissions (entry point, configuration.py, …)
#533
Configuration comes from an external python package instead of Rucio core
Python package needs to have a module for schema, permission, policy
Transparent handling of archives with rules
#1091
Move to next Feature release
Global Quotas
#2315
Making progress; Updating the CLI
PR there, needs review
Possibility to inject rules delayed
#2639
Improve oracle test crashes
#2588
Some improvements, unclear if it helped
Python 3.5 for server
pystatsd does not support Python3
Change to statsd possible
fts3-rest-api removed, myproxy dependency removed as well;
Postgres issue with byte and text data
Python 3.6 for clients
Should be compatible, but something missing in the tests
Changes for CTA transfer handling
#2632
Cedric and Martin will work on this next week
Move dev-container fts&xroot development over to travis container as well
Next test with CTA on their test instance
Source throttling
#2611
Needs merging
Activity exclusion for submitter
#2640
copy_pin_lifetime - Mario will work on it
staging times
JJ prepared PR for this
Also needs to be populated by conveyor poller
5
AOB
Minutes
Next week no meeting!
Nov 14: Rucio 1.22 release roadmap discussion