dCache

    11/24/2021 dcache pool umfs06_12 caused jobs to fail at staging-in files, restarting the dcache service resolved the problem.

    12/05/2021 dcache pool umfs23_1 caused jobs to fail at staging-in files, xfs_repair needed to resolve the problem.

    11/30/2021 Updated dCache from 6.2.32 to 6.2.35 to fix SRR report issue.
               The update was smooth. We also updated the firmware and kernel, and rebooted.
               The R740xD2 had new BIOS installed (2.12.2)

  Condor

    12/02/2021 We spotted some jobs nearly flooding one work node with a small disk/core (14GB),
               so we changed the max disk from 15GB to 13GB/core for the AGLT2 PanDA queue,
           this will stop reconstruction jobs from coming in.
           This is likely caused by a bug in condor 9.0.6 (schedule jobs to work nodes with insufficient disk space).
           ADC also mentioned they could work on reducing the intermidiate file sizes of the reconstruction jobs.

    12/06/2021 Did a rolling upgrade on condor from 9.0.6 to 9.0.8 to address a bug
               (Condor sends jobs to work nodes with insufficient disk space).
           The update went smoothly, We first did the work nodes without draining,
           and that requires setting a longer SHUTDOWN_GRACEFUL_TIMEOUT to 3 days
           to allow all remaining jobs to finish before condor restarts the StartD after the upgrade,
           however the condor_master itself does not get restarted.
           Then we did the sched nodes and head nodes and restarted the condor service after upgrading.

  Network

    12/04/2021 from Sat 12/04 1PM to Sun 12/05 3:30AM (13 hours)
               we lost the hard link between the UM and MSU sites.
           This was due to a hardware issue in the Merit service provider equipment.
           Replaced a DWDM card (dense wavelength-division multiplexing optical card in East Lansing)
           This meant the MSU site lost path to non-ESnet routes, including Merit DNS resolvers,
           but now have ACL access to MSU DNS resolvers.

  Hardware

   MSU & UM working on common quotes for R740xd2
    and R6525 with currently available AMD CPUs
    planning for about 50/50 storage/compute