We have had a LOT of troubles with score-dedicated WN since the upgrade to the foreshadow kernel 3.10.0-862.11.6.el7.x86_64 .  This was done simultaneously with an upgrade of HTCondor to 8.6.12 .  The symptoms are in a few different classifications:

The load suddenly starts to linearly increase, (this could be an artifact of some cron tasks) but the baseline seems to be that a lock of some kind on the file system is taken, but it is not released.  "top" shows no real work ongoing any longer.  So, cvmfs and HTCondor seem to argue about who should have access.  Commonly stopping HTCondor will clear all the race conditions.  Down-grading to HTCondor 8.4.11 is helpful, but issues still happen, and so the whole foreshadow kernel upgrade is suspect.

Another related symptom is that usage of swap suddenly surges to use all available.  No processes seem to be killed via oom, but again, load rises, and no real processing goes on.

Often the Condor startd goes unresponsive, and is killed by the HTCondor watch dogs, but that does not necessarily clear the issue.  HTCondor will sometimes totally crash out and stop, which usually results in an idle WN that must be restarted.

In a few cases, cvmfs was stuck in a tight loop trying to access disks, or cache, or....  Jakob put out a pre-release of 2.5.2 that addressed this too-aggressive behavior.  This helped, but did not resolve anything, and his final conclusion was that cvmfs itself was not at primary fault.

The problem seems to have no solution that I'm aware of.  We _could_ back down out of the foreshadow kernel....

We had a fiber cut on our private network a week or so back, with dropped half the HTCondor slots out of production.  This was repaired after about 16 hours or so.  At the same time, two of our internal switches quit talking, and this crashed our VMWare infrastructure.

Beyond this, we have been proceeding along fine.