Speaker
Description
Astronomical satellites serve as critical infrastructure in the field of astrophysics, and data processing is one of the most essential processes for conducting scientific research on cosmic evolution, celestial activities, and dark matter. Recent advancements in satellite sensor resolution and sensitivity have led to petabyte (PB)-scale data volumes, characterized by unprecedented scale and complexity, posing significant challenges to data processing. However, traditional data processing methods are facing the issues including intricate interdependencies among multi-level data products (e.g., Level 0 to Level 2), limited memory resources, and high memory occupancy rates, which collectively affect data processing efficiency. To address these issues, this study proposes a performance optimization framework for astronomical data processing. Firstly, an adaptive data chunking model is established, which realizes the data partitioning dynamically based on real-time memory availability and computational load. Secondly, a multi-level memory management method is presented, which optimizes memory utilization through caching the frequently accessed data into memory and building the priority-based queuing mechanism. Finally, a parallel data processing interface is introduced, which is developed to transform the algorithm from single-threaded serial execution to parallel processing. Experiments are conducted to verify the availability and practicality of the proposed framework, and the result shows that the data processing efficiency has been improved by 20%, effectively solving the deficiencies in traditional methods. The research outcomes will be implemented in the data processing tasks of the enhanced X-ray Timing and Polarimetry (eXTP) satellite, while also providing guidance for the data processing workflows of other astronomical satellites.
| Experiment context, if any | enhanced X-ray Timing and Polarimetry，eXTP |
|---|
