Packet marking
- CMS enabled scitags in Rucio production !
- FTS enabled scitags for all experiments in production !
- We were able to confirm that all this works, i.e. scitags are propagated all the way to the storages (Rucio -> FTS -> http-tpc -> Xrootd) works fine
- EOS enabled scitags in production for almost the entire CMS cluster (80%) - this works fine !
- Since xrootd is unable to filter local subnets/prefix, we had to setup a middle-service (collector) to catch the fireflies and forward them to ESnet (this also worked fine, but not on the first attempt)
- The additional benefit of this is that we're able to see/compare fireflies sent/received (and for EOS it matches)
- ESnet collector demonstrated capability to handle 15M fireflies/hour (during DC the rate from EOS CMS had peaks of ~ 100k/hour) !
- Firefly dashboard shows fireflies from UNL and CERN at
- https://public.stardust.es.net/d/b8dddac0-5b24-4739-9c8d-e88a05c1344f/scientific-network-tags3a-rande-dashboard?orgId=1&from=now-12h&to=now
- Dashboard will be updated to allow filtering per src_site (update sent to Andy, waiting for it to be deployed)
- Andy added median/max duration flow - looks very interesting - no "fat" flows from EOS CMS during DC !
- Cross-validation with other dashboards to be done
- Issues:
- We hit an issue at UNL with XRootd that causes it to crash while parsing http-tpc, this is fixed in XRootd 5.6.7
- We notified sites (mostly in UK) to disable fireflies until they're able to upgrade to XRootd 5.6.7.
- Unfortunatly this had a negative impact on firefly collection for UK sites during DC, but keeping transfers operational during DC has higher priority.
- We have also postponed ATLAS Rucio scitags production, but it's up from today.
- UNL is back in production running on 5.6.7 but reported an issue with some thread getting stuck in a loop