Apologies again; Katy is at Networkshop in Nottingham giving a plenary this afternoon. 

There was a red spot in the SAM tests last Wed on the AAA and Jyothish thought this was related to new VMs for the service he created in proxmox interfering with the federation. 

Brian pointed out a few files repeatedly failing to go to Antares from Echo. Looking closely, this turned out to be just 4 files, with the failures starting in second week of May. The issue was that the checksum saved on Echo was incorrect. However the file itself was correct when downloaded. I let Jyothish know in case it was related to streamed checksums. I also created a ticket for CMS DM to invalidate the 4 replicas on Echo so that Rucio can re-transfer.

Jyothish now has access to the recycled Antares machines which will be the new AAA proxy machines. He is adding an XCache and we will see if we can improve performance for jobs outside RAL reading data from Echo. 

Reminder to self: Still to look at the /unmerged/ problem.