Speaker
Description
In this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for
storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large
amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was
to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO
data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis
data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format
is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data
analysis frameworks to use.
In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with
more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.
Significance
The developed data format enables nuclear physics experiments to adopt a unified data format throughout the entire data lifecycle. It also supports column-wise analysis, catering to both physics and AI analysis workflows. The columnar data storage achieves over 3 times higher data throughput compared to ROOT and Parquet.
References
https://arxiv.org/abs/2501.07666
| Experiment context, if any | CLAS12 Experiment at Jefferson Lab |
|---|