Long term: UM server room needs refresh of power and UPS infrastrucutre
while UM is also considering a new data center on campus.
Short term: The R760xd2 we would like to purchase don't fit in the racks.
Mid term solution: replace 2 racks with deeper/wider racks
One LSA rack was replaced and added to AGLT2
The other was an existing AGLT2 rack, replaced July 24
No site downtime.
11 worker nodes were drained.
Moved 8 dcache servers, one at a time.
Those pools have been rdonly for over 24h.
MSU had a cached copy of the “hot files.”
Relocated the 8 storage servers in 2 hours
Small number of jobs (~110 jobs) failed due to data access failure.
Blacklisted due to Worker Node configuration problem, for 17h, July 27-28
Ansible policy to mitigate threats (preventing user namespace) had been mistakenly extended to the WNs.