May 25 – 29, 2026
Chulalongkorn University
Asia/Bangkok timezone

An Adaptive Serialization and Compression Optimization Framework Based on Data Feature Perception

May 26, 2026, 3:15 PM
1h
Sala Phra Kieo, Second Floor

Sala Phra Kieo, Second Floor

Board: 326-Tu
Poster Presentation Track 9 - Analysis software and workflows Poster

Speaker

Ms 世园 符 (高能物理研究所)

Description

Large scientific facilities such as the High Energy Photon Source (HEPS) generate massive volumes of heterogeneous experimental data during operation. These data exhibit remarkable diversity in terms of scale, structure, and distribution characteristics, imposing extremely high requirements on the real-time response capability and long-term archive storage efficiency of data processing systems. As a critical link in the data processing pipeline, data serialization and compression directly affect the overall efficiency of the system. However, existing methods typically adopt fixed algorithmic strategies, which are difficult to dynamically adjust for different data features, resulting in suboptimal performance when handling diverse data.
To address the above issues, this paper proposes and implements a data-driven serialization and compression optimization framework named LightPacker. The framework adopts a modular design, consisting of a data feature extraction module, a historical knowledge base construction module, a real-time decision-making module, an algorithm execution module, as well as an extensible algorithm library and an empirical knowledge base. It can automatically select and execute the optimal combination of serialization and compression algorithms based on the features of input data. The framework embeds a rich set of serialization and compression algorithms, establishes unified algorithm interface specifications, and defines integration standards for new algorithms, facilitating subsequent expansion. The feature extraction module constructs data feature vectors by analyzing multi-dimensional attributes such as data metadata, structural features, information entropy distribution, and numerical sparsity, and calculates their similarity with historical data. The historical knowledge base construction module conducts comprehensive performance evaluations of algorithm combinations on accumulated historical datasets, establishes mapping relationships between feature patterns and optimal algorithm combinations, and forms an empirical knowledge base to support rapid decision-making. For real-time input data, the decision-making module first retrieves the matching optimal combination from the empirical library based on feature similarity; if no matching record is found, it performs lightweight performance evaluations on candidate algorithms to quickly screen out potential optimal solutions. Finally, the execution module invokes the selected algorithm combination to complete the data processing task. To balance decision-making accuracy and system overhead, the framework adopts optimization strategies such as data sampling and caching mechanisms, effectively controlling additional computational costs.
Experimental evaluations under typical application scenarios of HEPS demonstrate that LightPacker achieves significant performance improvements in two core tasks: data transmission and data storage. Compared with traditional fixed-strategy methods, it can effectively reduce network transmission bandwidth consumption, accelerate the data preprocessing pipeline, and decrease storage space requirements. This framework provides an efficient, flexible, and extensible solution for data processing systems in large scientific facilities and similar scenarios.

Authors

Mr 誉 胡 (高能物理研究所) Ms 世园 符 (高能物理研究所) Mr 浩凯 孙 (高能物理研究所) Mr 建利 刘 (高能物理研究所) Mr 磊 王 (高能物理研究所) Mr 点 刘 (高能物理研究所)

Presentation materials

There are no materials yet.