Speaker
Description
Future high-energy physics and gravitational-wave experiments are projected to generate data at unprecedented event rates, demanding fast, scalable, and efficient data management systems. Because scientific data volumes continue to outpace available storage infrastructure, experimental workflows increasingly rely on a combination of real-time triggering mechanisms to filter uninformative events and lossless codecs to minimize the footprint of retained data. Neural compression methods have recently demonstrated significantly higher compression ratios than conventional codecs—such as LZ4, Zstandard, and dictionary-based encodings—by more effectively learning the underlying data entropy. However, these neural approaches typically incur substantial computational overhead during both training and inference, limiting their deployability in high-throughput production environments.
While neural compression has achieved remarkable success across mainstream modalities (text, images, and audio), recent efforts have successfully extended neural entropy models to scientific data, including lossless compression frameworks based on state-space models and Asymmetric Numeral Systems (ANS) for CERN datasets. Yet, while large sequence models can capture complex entropy distributions far more accurately than traditional frequency-table-based codecs, their extreme computational cost and long latency profiles often render them impractical for real-time scientific data ingestion.
In this work, we introduce a scalable neural compression framework based on a Byte Latent Transformer (BLT) optimized for the lossless compression of scientific event data. By operating on dynamically patched byte representations, our approach accelerates inference latency while maintaining competitive compression ratios, making it uniquely suited for line-rate data acquisition systems.
We integrate this compressor into a unified "filter-and-compress" pipeline, where incoming events are first triaged by an anomaly detection algorithm. Based on the computed anomaly score, an event is either rejected, compressed, or routed to a specific tiered storage destination. Because the entire framework operates natively on raw byte-level representations, it remains completely agnostic to the underlying data schema, seamlessly supporting heterogeneous, variable-length event streams. This end-to-end byte-level approach provides an efficient, scalable solution for selective storage and high-ratio lossless compression in next-generation scientific computing infrastructure.
| Do you plan to submit a 4-page extended abstract on OpenReview (only for Presentations/Posters)? | Maybe |
|---|