Site is now full of jobs and operating well

Updated site to OSG 3.3.24

Retirement of several storage nodes in dCache

Illinois Campus Cluster lost hypervisors

• Building which houses the ICC, ACB (Advanced Computation Building) had scheduled power outage 5/20
  • Upgrades to power feed to building
  • All equipment was powered off for 6 hours
  • First power down for many items in over 8 years
• Cold start caused various hardware failures
• Hypervisor cluster would not restart causing loss of MWT2 VMs
• VMs migrated to campus base hypervisor which was planned to happen in a few weeks
  • MWT2 VLAN had been extended to campus hypervisors
  • Migration took several days
  • VMs now use NFS to access GPFS due to VLAN outside ACB
  • Took time to reconfigure VMs to new setup
  • Some tuning still needed but all is working well

USERDISK down to only 12TB in use

SCRATCHDISK deletion still is issue

• Deletion is using rucio space reporting which seems to have a lot more free than dCache
• Lincoln/Judith working with Hiro/Armen