Speaker
Description
We present a novel integration of the PanDA workload management system (PanDA WMS) and Harvester with Globus Compute to enable secure, portable, and remote execution of ATLAS workflows on high-performance computing (HPC) systems. In our approach, Harvester, which runs on an external server, is used to orchestrate job submissions via Globus Compute’s multi-user endpoint (MEP). This MEP provides a function-as-a-service interface to the HPC resources, eliminating the need for custom, site-specific gateways while ensuring dynamic runtime configuration and secure access, without direct shell logins.
Using NERSC’s Perlmutter as our initial testbed, we have successfully deployed PanDA pilot jobs requiring shared services such as CVMFS for ATLAS software distribution and configuration management. The integration efficiently addresses key challenges including dynamic resource provisioning, runtime environment setup, and secure multi-user operation on HPC edge systems. We discuss our design decisions, the benefits of using a Globus Compute MEP, and strategies for mitigating configuration and dependency issues. Our results on Perlmutter and other HPC clusters demonstrate that our integration effectively supports the ATLAS simulation workloads with minimal overhead and performance portability. Our experiences further highlight the value of integrating PanDA Harvester with Globus Compute for portable HPC execution, demonstrating how this approach provides a scalable, adaptable, and secure remote execution solution that can be broadly applied across diverse scientific workflows beyond ATLAS.
Significance
This integration marks a significant step in executing HEP experriment workflows with high-performance computing (HPC) resources. This work addresses challenges that have limited HPC adoption in data-intensive experiments, including the heterogeneity of HPC environments and security requirements. As a result, complex workloads beyond ATLAS can now run on supercomputers with minimal custom adjustments, effectively making HPC resources as accessible and portable as traditional grid or cloud resources within the PanDA ecosystem.
| Experiment context, if any | This research is contextualized by the ATLAS workflow, where the workflow involves detailed simulation and analysis components integral to particle physics investigations. |
|---|
