Speaker
Description
As the scale and complexity of high-energy physics computing grows, storage systems are being pushed to serve radically diverse workloads at once, often with significant performance consequences. To ensure EOS can meet these evolving demands, we introduce a real-time I/O traffic-shaping framework that monitors ongoing I/O patterns and dynamically adjusts and balance read/write flows to maintain stable, efficient throughput, ensuring critical workflows continue to be served reliably even under heavy contention.
The concept introduces a lightweight, distributed I/O monitoring layer that collects detailed metrics directly from each storage node. These local measurements are periodically aggregated by the MGM namespace service, where they are analysed to compute bandwidth usage, IOPS distribution, and emerging congestion patterns. From this global view, shaping parameters - such as rate limits per application or per user, and prioritization hints - can be dynamically computed and propagated back to the nodes
This architecture forms the basis for a closed loop in EOS: monitoring, aggregation, and shaping work together to regulate throughput, improve fairness, and protect interactive responsiveness under load. The implementation uses open standards and efficient serialization formats (e.g., Protocol Buffers) to enable easy communication and future integration with machine learning or policy-driven optimization frameworks.
Ultimately, the real-time I/O shaping framework aims to transform EOS from a reactive storage system into an adaptive, self-regulating storage service capable of maintaining performance and predictability across diverse and fluctuating workloads.