Echo mini-DC was last week 3-7 March. Analysis is in progress. Talks at LHCONE/OPN meeting next week, then more at GridPP53

Monday - planned to only half-fill the OPN but almost filled it (200Gbps).

Tuesday - cut of the OPN and fallback to the LHCONE. A slight drop observed but actually it worked pretty well. 

Wednesday - CMS injections crashed on tuesday evening at 8pm. All VOs restarted the injections in the morning, drawing data from CERN and Tier 1s. Increased the rate to saturated the link and study T1->T1 links. Gateways started to struggle as they were saturated to 25Gbps with some 

Thursday - read tests from Echo. Didn't really intend to push it hard but it also ~saturated on the OPN.

Friday - Tier 2 tests cancelled/postponed due to major sites Lancaster and Manchester not being fully functional. Instead we did checksum-on-the-fly testing. This went well in terms of stress testing the code but there were errors with the checksums themselves (many of them were trying to checksum a file on read instead of only on writes, but there were also possibly a few genuine errors.

At times even though FTS traffic dropped to 100Gbps, we still see nearly 200Gbps in the network plots. So we suspect streaming from CERN from CMS and LHCb...investigating. James A created new plots showing traffic to/from storage and WNs (but the WNs show internal data movements too). Various monitoring from CMS and LHCb show 

Deletions: CMS was behind in deletions before the mini-DC started (and had gone over pledge). So deletions were going to struggle during the mini-DC. Katy got additional Rucio reaper pods started up to focus purely on RAL. Although the rate-per-file of deletions is clearly slower than some other sites, RAL seemed able to keep up with the higher rate of deletions once it was being targeted with continuous deletion requests.