Starting on the 2nd the VP queue was receiving very few jobs, with almost no files accessed in the xcache.  As of last night we are getting some jobs again, though the number of file accesses recorded in the xcache monitoring is still fairly low compared to what it was before.  Hammercloud jobs continued to be submitted and to run successfully, so the queue was never blacklisted.  The xcache seems to have been working throughout this time, the various monitorings all show green.  Unclear what was stopping jobs from picking the queue, any insight from ADC would be welcome.

On the 23rd we had to restart the cluster's CVMFS daemonset, as CVMFS access problems were causing many jobs to fail.  This resulted in a loss of running jobs, but we have seen no CVMFS issues since then.

On the 3rd we were briefly blacklisted to a storage issue.