<p>05/12/2022</p>

<p>Updated condor from 9.0.11 to 9.0.12</p>

<p>Updated gratia probes on all gatekeepers. Gratia probe stopped working for a day after the upgrade, and it was fixed by reconfigure, and then manually restart condor-ce and&nbsp; run&nbsp;</p>

<p>su - condor -c " /usr/share/gratia/htcondor-ce/condor_meter"</p>

<p>&nbsp;</p>

<p>05/17/2022</p>

<p>We migrated the Tier2 NFS server umfs02 to a virtual machine without having downtime. This nfs server provides the home directory for all grid users. The migration hit some problems: 1) the MSU work nodes could not mount the new NFS server because of routing issues . We added the routing rules as a workaround. 2) This nfs server also serves as the archive directory for the dCache postgresql databases’ hot standby replication. For one of the database servers (head01), the hot standby replication did not have a smooth transition during the 20minutes downtime when the NFS servers were swapping, so we ended up reseeding the database from head01 to its hot standby server d-head01.</p>

<p>&nbsp;</p>

<p>We converted all 26 remaining SL7 servers at UM site to CentOS7, this includes all the dCache pool nodes and lustre storage nodes.&nbsp;</p>

<p>&nbsp;</p>

<p>05/21/2021</p>

<p>The new nfs server(virtual machine) umfs02 lost accessibility, increasing the memory and CPU restored the service.The site drained to 10% usage on 21st because of this incident.&nbsp;</p>

<p>&nbsp;</p>

<p>05/23/2022</p>

<p>Gratia on the OSG gatekeeper (gate02) stopped working for 2 days. Restarting condor-ce service fixed that</p>

<p>MSU finished installing and phasing in the 3x new VMware AMD host nodes (ordered Sept 2021).<br>
But still using old direct-attach SAS storage. Last step will be to start using new NVMe storage via iSCSI (also received from 2021 order)&nbsp;</p>