SWT2_CPB:
Campus Facilities Management performed electrical work and transitioned the data center from building to generator power from 7/24-7/25.
We took time to prepare for this by reducing our power usage (by partially draining and powering off our oldest worker nodes, plus other nodes that would not affect the production cluster), providing additional battery runtime on our UPS and reducing temperature in the room.
There were no issues. We never lost power, the transition on and off the generation went smoothly, and we returned all worker nodes to their previous state on 7/31.
We set the SWT2_CPB_K8S offline, drained K8S worker nodes, and plan on rebuilding these nodes to be used in the SWT2_CPB PQ.
There may have been a potential issue in the XRootD Redirector configuration that may have caused errors in the SWT2_CPB_K8S PQ. The change was made during implementing a second redirector, but we are unsure. Jobs started to succeed right before draining after reverting this change.
Experienced memory leak issue with all four of our DTNs.
Due to 330k certificate CRLs being revoked by HARICA CA, causing memory/space issues on GRID middleware.
We implemented jemalloc, which significantly improved memory issues on all of our DTNs. We have not had any XRootD Proxy service fail so far after this change.
We also modified the service to automatically restart (with limits) and to have OOM stop the processes more cleanly.
Troubleshooting and diagnosing this issue last week, we contacted XRootD experts, tested XRootD version 5.9.6 in our test cluster, added additional memory, and tried other methods for minimizing or solving issues at our site.
Implemented additional monitoring and alerts of our XRootD Proxy servers.
GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation (SWT2_CPB)
Removed old certs, but did not work. Performing transfer tests after change failed. We reverted this change.
OU: