Ticket 1001213
Jobs failing. Writing new files was failing. Our storage appeared to be full.
That happened after too many pools were left set rdonly for too long.
The rdonly pools slowly drained (file system showing 66%) following DDM deletions
and the remaining RW pools slowly filled up (to 98%) with new files.
We had forgotten to set back RW half of the pools from one site
after the on-the-fly rolling dcache update 4 weeks earlier.
We already had a cron job alerting for pools becoming offline
and it has now been upgraded to also flag rdonly pools.
We also re-balanced all pools site wide.
To re-spread the unused space among all pools.
It is used for temporary cache between UM-MSU.
Condor/Condor-CE updates to OSG25
Condor on 25.0.3
Condor-CE on 25.0.1