A bad week for SAM tests:

CMS did not respect the tape downtime on webdav (only on xrootd). The downtimes are only declared 'natively' on xrootd, with webdav as a 'child'. CMS need both protocols to be fully declared for Antares. SAM tests were corrected - but I was informed this has happened before. 

CMS was also sensitive to Echo/gateway issues we are currently experiencing since Wed. CMS SAM tests have red days since Thursday until now. Katy brought CMS out of drain on Monday. CMS is back in drain again today.

For the new AAA proxy machine - now being tested by SAM tests. Katy still to add to monitoring in Vande (also Shoveler once we have the production instance).

AAA machines needed services restarting again. We had a ticket from a user unable to access data. 

The cms-rucio-services machine needs VOMS infrastructure removing. It appears the auth has already stopped. Katy has started approving tape transfers and defining tape families by hand. https://stfc.atlassian.net/jira/servicedesk/projects/STAH/queues/custom/263/STAH-91   UPDATE - temporarily fixed by updated the host to recognise use the CMS IAM VOMS server. This machine still needs upgrading etc. but will be done after other, more urgent tasks.