Discussion topics:
- Improve Submitter (an co.) performance (#6505)
- Also affects conveyor- preparer and stager
- Does not make sense to submit transfers for rules which are already expired
- Hotfix was made (for ATLAS), will be pushed upstream soon
- CMS:
- Submitting (datasets) blocks with big files-> was ok
- Submitting (datasets) blocks with small files -> saw this problem as well
- Improve Cleaner performance (#6511)
- Cleaner having hard time to clean ongoing rules
- Needs a small re-design of the cleaning workflow
- Prevent overlap between Reaper threads (#6512)
- Bulk-size configurable, however, last_updated time of replicas is not updated
- Update remaining replicas during the loop
- CMS:
- Lots of deletion not found errors due to this
- Manual solution: Partition reapers
- DUNE
- Channels clogged up so much due to misconfig/misunderstanding of configuration
- Ability to warn that one link is clogging the system
- Possibly do-able via the new /metrics endpoint
- Scaling of frontend httpd servers
- Worked well in handling the load
- pod config similar to ATLAS ones, running 12 pods with default mod_wsgi config
- Other daemons increased a bit as well (similar to ATLAS config)
- Timeout when calling metacat server via policy
- Not clear that it comes from metacat
- Standard concsistency tools (dark data, etc.) would be helpful
- Scaling of daemons/services
- Submission of lots of small files
- Scaled up 3 pods to 8 pods worked for CMS
- Kubernetes Auto-Scaling could solve this, but it is not trivial
- No clear recommendation how many pods/threads are needed for what load -> experimental mostly
- Rucio tries to delete replicas that were never transferred
- Clearly useful to avoid the creation of dark data, but affects deletion performance in artificial situations like this
- Replica never transferred -> replica goes to UNAVAILABLE -> goes to reaper
- Tokens
- Polling for about to expire tokens was biggest problem, something to adress in the next version of FTS
- Katy: Repeat a token specific mini challenge later in time
- Yes, new release of FTS should come in April
- Mini challenges will need to be done also for Rucio upload/download workflow
- #of refresh tokens stored in IAM issue needs to be adressed
- Alternative IdPs (CiLogon) needs to be tested as well
- IAM:
- Could there be a procedure to wipe DB (if clogged)
- Coarseness of token-scopes still hot topic
- CMS
- Rucio worked really well and the injector tool was excellent!