24–27 Sept 2019
EC-JRC Ispra
Europe/Rome timezone

HTCondor and containers for Batch and Interactive use: (Mostly) a success story

24 Sept 2019, 17:00
25m
EC-JRC Ispra

EC-JRC Ispra

European Commission – Joint Research Centre Via Enrico Fermi, 2749 I - 21027 Ispra (VA) Italy N45° 48' 36.09'' E008° 37' 16.72'' N45° 48.601 E008° 37.278 45.80998, 8.62135 https://www.openstreetmap.org/#map=17/45.80998/8.62135
HTCondor presentations and tutorials Workshop presentations

Speaker

Oliver Freyermuth (University of Bonn (DE))

Description

An HTC cluster using HTCondor has been set up at Bonn University in 2017/2018.
All infrastructure is fully puppetised, including the HTCondor configuration.
Both interactive and batch jobs are run inside Singularity containers, and users only have to choose the desired OS via a job parameter from an offered collection of container images without setting up or building a container themselves.

The container images are rebuilt daily and are provided by a CernVM filesystem (CVMFS) along with various software packages. The data to be analysed is stored on a CephFS file system.

This talk presents the successful migration from a "classic" PBS cluster setup with separate login nodes providing an environment similar to the batch environment, causing many regular admin headaches, to a modern and flexible, HTCondor-based system offering the users several different environments both for interactive and batch usage. Both the successes and the several pitfalls and issues which were encountered will be discussed.

Desired slot length 20
Speaker release Yes

Primary authors

Peter Wienemann (University of Bonn (DE)) Oliver Freyermuth (University of Bonn (DE))

Presentation materials