SWT2_CPB:
We rebuilt five R740 storage servers from EL7 to EL9 while preserving data.
After the rebuilds, we verified the data and it appears to have been preserved. We have a temporary backup of the data in case of any data loss.
The servers have been returned to production, and we have not observed any issues so far.
We are currently creating backups of additional R740 storage servers for migrating them from EL7 to EL9.
We changed the certificate on one of our XRootD Proxy servers to make it a full-chain certificate. All of our XRootD Proxy servers now have full-chain certificates.
We are monitoring for any new or increased errors such as with SSL Handshake Exception.
11.9% of our production jobs have failed within 12 hours on 3/30 (457 failed production jobs).
One worker node (compute-19-39) showed 1% of jobs succeeding (6 succeeded, 378 failed).
Removed this node from service for now to investigate. Ensuring there is not an issue with the worker node before returning putting it back into production.
Specific Errors:
Failed to execute payload:PyJobTransforms.transform.execute CRITICAL Transform executor raised TransformValidationException: EVNTtoHITS got a SIGSEGV signal (exit code 139)
Non-zero exit code from transform substep executor
The errors started to increase at 2:20 a.m. UTC 3/30. So far, the peak was at 6:00 a.m. UTC, and lowered back to down at 8:40 a.m. UTC 3/30.
OU: