Speaker
Description
The LHCb Online Mover is a critical component of the LHCb online computing stack, responsible for streaming data accepted by the High Level Trigger 2 (HLT2) from online storage to long-term offline infrastructure. During data-taking, data is produced at sustained rates of up to 20 GB/s, with bursts reaching 50 GB/s. For efficient long-term storage, the data must be compressed and packed into files. Sustaining these rates without interfering with trigger operations imposes strict CPU and memory constraints. Approximately 120 PB of data is processed per year and subsequently exported to EOS and registered in DIRAC for offline processing and analysis.
Zstd compression dominates CPU load, while DIRAC requires files of 5–10 GB. These files can only be transferred via XRootD at approximately 200 MB/s. To meet the overall throughput requirements, several such files must therefore be transferred in parallel, requiring temporary buffering while new files are produced concurrently. SSDs cannot withstand the required write endurance and HDDs are too slow, making RAM the only viable buffering medium. Minimising the memory footprint is therefore essential for scalable operation.
In this work, we present a newly redesigned Online Mover architecture based on Rust. The new design unifies data handling and compression pipelines, employs efficient multithreading, and introduces asynchronous I/O for network transfers. Several design variants were benchmarked in terms of throughput and memory usage. The optimal solution achieved a 90 % reduction in RAM consumption while increasing total throughput by 15 %, enabling a more scalable and cost-efficient Online Mover service.