SWT2_CPB -

• Site not full for a couple of days. The site has been draining and refilling repeatedly. We suspect A possible change to the CVMFS wrapper, which is causing our compute nodes to fail jobs due to missing CVMFS mount. We are still trying to investigate this though.

• We also suspect DC24 may have caused issues for our site prior to the CVMFS issue. 

• Created script to modify NHC config file to check for missing mount and pull node out of Slurm in order to alleviate CVMFS issue for now.

• UPS upgrade is scheduled for Thursday 3/7 (tomorrow).  

• De-bugging hardware problems on a few compute nodes - can be tedious

• Working on resolving perfSONAR issues.

• Working on testing and implementing storage node token. 

• Continuing to test Google HIMEM queues.

OU:

• Generally stable operations.
• Occasionally deletion timeouts which are not understood. Some are caused by temporary storage overload, but others happen without load, so it appears that rucio deletion attempts of non-existing files take a long time to return 'file-not-found' the first time, and then successive identical requests are faster. Need to work with Andy and Wei to traci this down.