23–27 Sept 2024
Nikhef
Europe/Amsterdam timezone

Practical experience with an interactive-first approach to leverage HTC resources

27 Sept 2024, 09:25
20m
Colloquium room (Nikhef)

Colloquium room

Nikhef

Nikhef Science Park 105 1098 XG Amsterdam
HTCondor user presentations Workshop Session

Speaker

Oliver Freyermuth (University of Bonn (DE))

Description

Development and execution of scientific code requires increasingly complex software stacks and specialized resources such as machines with huge system memory or GPUs. Such resources are present in HTC/HPC clusters and used for batch processing since decades,but users struggle with adapting their software stacks and their development workflows to those dedicated resources. Hence, it is crucial to enable interactive use with a low-threshold user experience, i.e. offering an SSH-like experience to enter development environments or start JupyterLab sessions from a web browser.

Turning some knobs, HTCondor unlocks these interactive use cases of HTC and HPC resources, leveraging the resource control functionality of a workload manager, wrapping execution within unprivileged containers and even enabling the use of federated resources crossing network boundaries without loss of security.

This talk presents the positive experience with an interactive-first approach, hiding the complexities of containers and different operating systems from the users, enabling them to use HTC resources in an SSH-like fashion and with their JupyterLab environments. It also provides a short outlook on scaling this approach to a federated infrastructure.

Desired slot length 20
Speaker release Yes

Authors

Michael Hubner (University of Bonn (DE)) Oliver Freyermuth (University of Bonn (DE))

Presentation materials