The 'fetch-crl' command went missing (https://stfc.atlassian.net/l/cp/01CAs1Gv) and many CRLs expired on Sunday about 5pm. Failures in the SAM tests on all CEs, Echo webdav (not xrootd), Antares xrootd (not webdav) and all the AAA machines. Echo webdav was partially fixed on Sunday around midnight, with intermittent failures after that. Everything was fixed before lunch on Monday. 

About 750TB of CMS data for Antares was in backlog due to a CMS WM bug. Rules that should have been distributed over the previous 1 month were created for approval on Wednesday afternoon and about 2000 of them were approved on Thursday, with the data starting to hit Antares from Thursday night. On Friday afternoon the remaining 2500 rules were approved. Some issues, for the record:

1. Errors of the type 'MGM is down' on Thursday night/Friday morning. George realised that a test setup had been accidentally deployed on prod. This was quickly reverted and those errors stopped. Rate to the Antares buffer went up to 15-20GB/s! (Worries about the buffer filling up, but George said it never did)
2. A large number of 'file exists' errors - these usually happen as a result of previous problems.
3. Friday evening - CMS DM report that Echo is in danger of going over pledge. Deletions are too slow to keep up with Rucio submissions for data multihopping through Echo (most of it was doing that). A bespoke Rucio-reaper is deployed to only work on RAL as we did during DC24 - this hugely improves the deletion rate. At the same time, the inbound transfers to Echo on FTS was reduced from 1000 to 200. Space used on Echo stabilised. The rate to Antares reduced greatly - possibly some of the 200 transfers to Echo were not also going to tape.
4. Transfers to Echo stopped on Sunday evening due to the fetch-crl problem mentioned above.
5. Monday morning - tape robot arms broken, fixed on Tuesday afternoon. Katy waiting for the backlog from multiple VOs to subside before trying an increase of the FTS inbound transfers to Echo again.

As a result of the above, SAM status was red on Friday, Sunday, Monday, Tuesday. Katy removed CMS from drain on Monday after the CRL was fixed. Interestingly, because other VOs started draining before CMS, CMS picked up many slots (35k at max!) before draining from that high value. Job performance remained good during this time. CMS went into drain again this morning (Wed), Katy removed the drain status before lunch.

The 'production' Shovler instance has been switched from Cloud to VMWare. VMWare is more resilient for a production service.

IPv6 connectivity of the AAA machines. A ticket has been sent to DI, which Katy cannot view but requested the service desk to progress it.