Speaker
Description
The HEP group at the University of Victoria operates a distributed cloud computing system for the ATLAS and Belle II experiments. The system uses private and commercial clouds in North America and Europe that run OpenStack, Open Nebula or commercial cloud software. It is critical that we record accounting information to give credit to cloud owners and to verify our use of commercial resources. We want to record the number of CPU-hours of the virtual machine.
To obtain the required information, we run a fast benchmark at boot time to give an estimate of the HEPSpec06 units of the node. Our first system, writes the benchmark and CPU times (obtained from /proc/stat) to a log file every 15 minutes. The last entry of the VM was used to determine the CPU-hours. This system has work well but the information about a VM is only available after it is deleted and, in some cases, VMs can exist for many weeks. Hence, the final accounting information is delayed for some time.
We have introduced a new system that continuously collects the information and uploads it into a an Elastic Search database. The information is processed and published as soon as it is available. The data is published in tables and plots in Kibana and Root. We have found the system to be useful beyond gathering accounting information and can be used for monitoring and diagnostic purposes. For example, we can use it to detect if the payload jobs are stuck in a waiting state for external information.
We will report on the design and performance of the system, and show how it provides important accounting and monitoring information on a large distributed system.
