• The cluster is running fine. At times there was a spike in stage-out/stage-in errors, which must be related to connectivity issues (ticketed in ggus:160124).
• Occasionally there are OOM termination of jobs running in old nodes with 2GB/core (never in the nodes with higher GB/core). Those nodes are request optimized to run at maximum efficient mode for job occupancy. When Request/Task is using much more memory than requested, the node may run out of memory and jobs terminated. Looking into a solution which may work for all type of nodes. As I was investigating this, I noticed that some panda pages (lookup from WorkerID) are not showing any job selection - pinged pandamon support.
• Trying to optimize the job CPU requests coefficient sent from Harvester (has 0.9 scale down value as default). The idea is to not overcommit the node CPU, but at the same time leave CPU request space for other system/auxiliary pods.
• Follow-up on the issue with the name of the parameter "resource_type_limits.SCORE_HIMEM" in CRIC, which got an extra space typo in the name. Fixed it for SWT2_CPB_K8S, and also pinged to Ryan (Victoria), so he fixed it for his site too, but for a while there was no feedback from the CRIC expert, so Fernando deleted that parameter from the CRIC. With some delay we got a response from Alexey. He checked DB that there are no more "bad" names, and they now also have mechanism in place to avoid introducing parameters with such typos.