Speaker
Description
Distributed computing infrastructures are shared by multiple research communities, particularly within High Energy Physics (HEP), where precise and transparent resource accounting is critical. To meet these demands, we developed AUDITOR (AccoUnting DatahandlIng Toolbox for Opportunistic Resources), a flexible, modular, and extensible accounting ecosystem designed for heterogeneous computing clusters. AUDITOR is deployed at major HEP sites such as CERN, KIT, where AUDITOR processes over 13 million jobs per month. This figure is expected to increase by a factor of 7-10 with the forthcoming High-Luminosity LHC era and AUDITOR is equipped to handle this increase in job counts.
AUDITOR captures, processes, and analyses usage metrics through specialized collectors from HTCondor, Kubernetes, and Slurm batch systems, storing all data in a central PostgreSQL database. AUDITOR is built in Rust, providing high performance and minimal memory footprint. Its plugin-based architecture allows integration with external tools via a REST API, with Rust and Python clients. Existing plugins include the APEL plugin, which publishes accounting data to the European Grid Initiative (EGI) and priority plugin, which sets the priorities of different groups on a batch system.
Recent developments include Role-Based Access Control (RBAC) system for fine-grained data access and an archival subsystem that now periodically exports historical data to Parquet files, optimizing database performance and enabling efficient long-term storage. The new utilization report plugin provides summaries of job counts, HEPScore performance, power consumption, and estimated CO₂ footprint. Looking ahead, our evolving accounting system is being designed to integrate environmental attributes directly into job-level analytics, enabling a deeper understanding of the energy efficiency and carbon impact of computing workloads. In this talk, we will present the architecture and capabilities of our new utilization reporting tool and highlight how these features are preparing us to meet the present and future challenges of accounting in high-performance and sustainable computing.