Speaker
Description
Workflow Management Systems (WMSs) are essential tools for structuring an arbitrary sequence of tasks in a clear, maintainable, and repeatable way. The popular Python-based WMS luigi helps building complex workflows. It handles task dependency resolution and input/output tracking as well providing a simple workflow visualisation and a convenient command-line integration.
The extension b2luigi, which is designed to be a drop-in replacement of luigi, offers easy integration with batch systems such as HTCondor, LSF, Slurm, and the WLCG, allowing the combination of heterogeneous tasks and systems within a single workflow. Furthermore, b2luigi provides additional interfaces tailored for interactions with the Belle II analysis software framework and the Belle II distributed computing tools.
In November 2023, the Belle II collaboration took over the development of b2luigi. Since then, several new features have been introduced, such as the ability to run tasks with Apptainer, the support for the Slurm batch system, and the capacity to define targets using the XRootD protocol. The documentation has also been extended, including a step-by-step tutorial covering all the main features. The b2luig package has become essential not only for physics analyses, but also for managing a variety of complex workflows. These include: software release validation, data reprocessing, detector calibration, and the derivation of systematic corrections for analyses.
In this contribution, we present an overview of the current status of the b2luigi project, highlighting recent developments, new features, and the deployment within the Belle II collaboration. Furthermore, we discuss the adoption and application of b2luigi beyond the Belle II collaboration, demonstrating its versatility and broader relevance to the high-energy physics community.
Significance
This presentation reports on the transition of b2luigi to official maintenance by the Belle II collaboration, marking a major step with the deployment of its first stable releases. Now widely used throughout Belle II for tasks ranging from physics analysis to calibration and data reprocessing, b2luigi has proven to be both reliable and versatile. Its simplicity and flexibility have also led to adoption beyond Belle II, demonstrating its value as a general purpose WMS.
References
- https://doi.org/10.5281/zenodo.10853220
- https://github.com/belle2/b2luigi/
- https://b2luigi.belle2.org
| Experiment context, if any | Belle II |
|---|
