Speaker
Description
Machine learning approaches have been widely adopted across several areas of high-energy physics research, including simulations, anomaly detection, and trigger systems. Deploying machine learning in trigger systems requires inference approaches capable of processing data at enormous rates, often on the order of 10–100 thousand events per second while making real-time decisions about which events to retain. This, in turn, demands inference pipelines that are both extremely low-latency and highly reliable. While ONNX Runtime provides a portable solution for inference, several applications require specialized libraries. Libraries such as NVIDIA’s TensorRT and AMD ROCm’s MIGraphX provide highly optimized inference stacks; however, integrating them into existing workflows often requires significant effort to convert input data, manage data formats, and handle library-specific configurations. Although specialized libraries serve specific purposes for particular applications, their integration and long-term maintenance remain challenging.
Code-generation approaches such as SOFIE can also enable efficient inference, but their integration into workflows can be cumbersome, particularly when model architectures or input formats change.
We present Yukti: a unified interface that enables inference across multiple target libraries without copying data or requiring configuration changes by the user. Acting as a wrapper, the interface allows input data to be referenced directly, avoiding data movement and additional user configuration. The interface also allows using the C++ code generated by SOFIE for inference in a load-and-use style. This design simplifies maintenance and helps ensure consistent inference performance as models evolve.
In its first iteration, the interface supports processing alpaka buffers, leveraging their abstraction for heterogeneous computing environments. Support for additional input formats can be added seamlessly based on user requirements. The interface consists of two main components: a processor, which runs offline to convert trained models into optimized plans tailored to the target inference library (e.g., converting an ONNX model into a .plan file for TensorRT, .mrt for MIGraphX, or .hxx in SOFIE), and a runtime interface, which loads these optimized plans and executes inference efficiently using the selected backend.
Finally, we present benchmarking results for several models executed through our interface across different inference libraries, highlighting their respective performance characteristics. By providing a single unified interface, our approach reduces workflow complexity, minimizes data movement, and ensures that updates to model architectures or input formats can be accommodated with minimal overhead. This enables developers to deploy machine learning models across heterogeneous hardware and software stacks more efficiently and with lower maintenance effort.