- Anomalous activity, caused by sPHENIX, on one of the SCDF NetApp appliances severely degraded the CVMFS Stratum-1 performance starting at midnight ET Tuesday morning. This lasted for ~10 hrs before the issue was identified and mitigations were put into place. During that time BNL and BNL_OPP queues were taken offline by HC for a few hours. Tier-2 sites were also impacted.
- We are in the process of deploying new hardware for the Stratum-1 (just recently received) and that will eliminate this kind of shared storage dependency going forward.