ACAT 2025

Design and implementation of JUNO Keep-Up Production pipeline

Not scheduled

30m

Hamburg, Germany

Poster Track 1: Computing Technology for Physics Research Poster session with coffee break

Speaker

Tao Lin (Chinese Academy of Sciences (CN))

Description

The Jiangmen Underground Neutrino Observatory (JUNO) is a multipurpose neutrino experiment with the primary goals of the determining the neutrino mass ordering and precisely measuring oscillation parameters. The JUNO detector construction was completed at the end of 2024. It generate about 3 petabytes of data annually, requiring extensive offline processing. This processing, which is called Keep-Up Production, typically involves multiple steps, such as preprocessing, calibration, reconstruction and data analysis. Automating this pipeline significantly enhances the efficiency of data production.

This contribution presents the design, implementation and application of JUNO Keep-Up Production pipeline that leverages Apache Kafka for inter-step communicate via messaging. This approach decouples the various steps, with each message containing metadata for a file, such as its name and path. When a task at one step is completed, a message is sent to a topic, notifying the subsequent step to initiate a new task.

To handle the distinct commands required at each step, a YAML-based job management tool has been developed. The tool comprises four microservices: an API server, a job creator, a job submitter and a job monitor. The job creator can be configured to generate jobs for individual files or batches. For processing a batch of files, the job creator caches files until the file list meets the specified requirements, allowing for flexible run-by-run data processing. Once a job is created, its information is registered with the API server, from which the job submitter retrieves and submits tasks. The job monitor tracks job statuses and, upon completion, generates a new message to trigger the next processing topic. Docker Compose is employed to create instances for these steps.

Finally, this contribution demonstrates the successful application of the Keep-Up Production in the JUNO experiment.

Significance

This Keep-Up Production pipeline is used in the JUNO offline data processing