Fix for the backoff issue is in the pilot. Observe very few backoff errors over the last few days, though still non-zero. (Also there are some jobs that throw the backoff limit error but are still marked as finished, not failed, which I don't understand.). Thanks to Fred and Michal for following up! Discussion is continuing on GGUS and JIRA.
Tape downtime due to NESE tape system maintenance.
Issue where a pool that had IO blocked due to a kernel panic caused a lot of job failures.
Blacklisted the weekend after Thanksgiving, slow recovery due to HC test 1293 not submitting a new job.