Speaker
Description
PanDA executes millions of ATLAS jobs a month on Grid systems with more than
300k cores. Currently, PanDA is compatible only with few HPC resources due to
different edge services and operational policies, does not implement the pilot
paradigm on HPC, and does not dynamically optimize resource allocation among
queues. We integrated the PanDA Harvester service and the RADICAL-Pilot (RP)
system to overcome these disadvantages and enable the execution of ATLAS,
Molecular Dynamics and other workflows on HPC resources.
Harvester is a commonality layer which brings coherence to diverse HPC
systems, providing integration with PanDA workflows at job and event level. RP
is a pilot system capable of executing short/long-running single/many-cores
tasks on diverse HPC machines, supporting CPUs, GPUs, and multiple MPI
implementations.
We integrated Harvester and RP on Titan at ORNL, prototyping a Next Generation
Executor (NGE) to expose RP capabilities and manage the execution of PanDA
workflows. RP acquires Titan resources via queues and backfill capabilities
and publishes the available resources to NGE. Harvester requests available
resources and submits tasks for execution to NGE. NGE uses RP to execute those
tasks, managing input and output staging, and holding the states of resources
and tasks on a dedicated database.
