- At the T1 we will be retiring 3000 job slots provided by Nehalem based WNs procured in 2009. These nodes will stay in the data center but will be re-purposed as resources managed under OpenStack.
- We have started preparing the FY16 disk storage procurement
- The storage management group is in the process of creating the inventory dumps requested by ADC. The process is delayed because of the ongoing deployment of the new storage hardware.
- We've noticed significantly higher than normal I/O (not CPU) load on our CVMFS proxies starting on December 2 (see attached plot)..From some debugging on the farm with systemtap, it appears that the issue may be caused by jobs in the BNL_PROD_MCORE queue. It seems from Wednesday until Saturday morning (in the US) there were a number of jobs running in this queue which have spawned short-lived shell scripts and python code such as miniAthFile.py. After some additional digging, it appears these may be Pile jobs, like [1]. Shortlived processes in the analysis queues are also generating accesses. In general, each time these shortlived processes are spwaned, we're seeing a significant amount of CVMFS access, likely due to LD_LIBRARY_PATH, and PATH settings, as well as the actual loading of the libraries/software involved in the execution.