Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

Integration of grid and local batch system resources at DESY

Oct 13, 2016, 2:45 PM
Sierra B (San Francisco Mariott Marquis)

Sierra B

San Francisco Mariott Marquis

Oral Track 6: Infrastructures Track 6: Infrastructures


Andreas Gellrich (DESY)


We present the consolidated batch system at DESY. As one of the largest resource centres DESY has to support differing work flows by HEP experiments in WLCG or Belle II as well as local users. By abandoning specific worker node setups in favour of generic flat nodes with middleware resources provided via CVMFS, we gain flexibility to subsume different use cases in a homogeneous environment.
Grid jobs and the local batch system are managed in a HTCondor based setup, accepting pilot, user and containerized jobs. The unified setup allows dynamic re-assignement of resources between the different use cases. Furthermore, overspill to external cloud resources is investigated as response to peak demands.
Monitoring is implemented on global batch system metrics as well as on a per job level utilizing corresponding cgroup information.

Primary Keyword (Mandatory) Computing facilities

Primary authors

Andreas Gellrich (DESY) Birgit Lewendel (Deutsches Elektronen-Synchrotron (DE)) Christoph Beyer (DESY HH) Thomas Finnern (DESY) Thomas Hartmann (Deutsches Elektronen-Synchrotron (DE)) Yves Kemp (Deutsches Elektronen-Synchrotron (DE))

Presentation materials