AGLT2 in downtime with 2 separate issues:

Another unexpected problem with transition of dCache spaceManager node head02 from EL7 to EL9. 
We now use version specific installs like postgresql15 and then use alternatives to pick the default version.
But ansible had not been updated to match this new method, and ansible updated the default version 13 provided by RHEL9 which broke the version selection.
This must have also been the source of the problem with backup/standby d-head02 that caused
the earlier replication problems and, indirectly, the intermittent outages.

While the replication node was offline and not processing files, one disk space was filling up.
Among other measures, we added some cleanup code to a script to delete some duplicated unrelated accounting files.
But, under some unexpected condition, the script could instead try to delete everything.
This dramatic misbehavior occured on a subset of nodes where this script runs regularly.

Current AGLT2 atatus and recovery:

We have declared a downtime that we will extend.

We have recovered head02 with a clean RHEL9 re-instal and using the last database backup from 06-Dec.
So there will be a loss of name records.

We are using this downtime to also upgrade/update head01 to RHEL9 and postgres15, then postgres17.
And we will also update head02 to postgres17.

We had a number of nodes that were damaged beyond repair and need rebuilding, including the DNS nodes at UM.

We are also considering upgrading the Condor nodes to RHEL9, but probably not yet OSG24.
We would benefit from the MWT2 knowledge and previous experience.