Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Consolidating the Grid and interactive analysis infrastructure at DESY - past and future

Nov 7, 2019, 3:30 PM
Hall F (Adelaide Convention Centre)

Hall F

Adelaide Convention Centre

Poster Track 7 – Facilities, Clouds and Containers Posters


Thomas Hartmann (Deutsches Elektronen-Synchrotron (DE))


DESY manages not only one of the largest Tier-2 sites with about 18 500 CPU cores for Grid workloads but also about 8000 CPU cores for interactive user analyses. In this presentation, we recapitulate the consolidation of the batch systems in a common HTCondor based setup and the lessons learned as both use cases differ in their goals. Followingly, we will give an outlook
on the future developments.\newline
While for Grid jobs startup latencies are negligible and the primary focus is on an optimal utilization of the resources, users of the {\it National Analysis Factory} for interactive analyses prefer a high responsiveness of the batch system as well as the storage.\newline
In the ongoing evolution of the batch system, we are exploring two different approaches to abstract the batch node's host OS from the actual job OS. For Grid jobs we are running legacy workloads in lightweight Singularity containers deployed via CVMFS. For interactive NAF jobs, we move towards Docker containers for a more heavy weight replication of a full batch node to provide the users with their full set of accustomed tools.\newline
To utilize resources best we further investigate the opportunistically backfilling resources, especially GPU nodes, between pools without interfering with user experiences.

Consider for promotion No

Primary authors

Thomas Hartmann (Deutsches Elektronen-Synchrotron (DE)) Patrick Fuhrmann Volker Guelzow (Deutsches Elektronen-Synchrotron (DE)) Yves Kemp (Deutsches Elektronen-Synchrotron (DE)) Thomas Finnern (DESY) Christoph Beyer Andreas Gellrich (DESY) Martin Flemming (DESY) Birgit Lewendel (Deutsches Elektronen-Synchrotron (DE)) Dr Christian Voss (DESY)

Presentation materials