9-13 July 2018
Sofia, Bulgaria
Europe/Sofia timezone

Dynamic Integration and Management of Opportunistic Resources for HEP

12 Jul 2018, 11:15
Hall 10 (National Palace of Culture)

Hall 10

National Palace of Culture

presentation Track 8 – Networks and facilities T8 - Networks and facilities


Matthias Jochen Schnepf (KIT - Karlsruhe Institute of Technology (DE))


Experience to date indicates that the demand for computing resources in high energy physics shows a highly dynamic behaviour, while the provided resources by the WLCG remain static over the year. It has become evident that opportunistic resources such as High Performance Computing (HPC) centers and commercial clouds are very well suited to cover peak loads. However, the utilization of this resources leads to new levels of complexity, i.e. resources need to be managed highly dynamic, HEP applications require a very specific software environment usually not provided at opportunistic resources and network bandwidth limitations can cause I/O-intensive workflows to run inefficiently.

The key component in order to successfully run HEP applications in a dynamic fashion on opportunistic resources is the utilization of modern container and virtualization technologies. Based on these technologies, the Karlsruhe Institute of Technology (KIT) has developed the resource scheduler ROCED to dynamically integrate and manage a variety opportunistic resources. In combination with the HTCondor batch system, a powerful single entry point to all available computing resources has been created, leading to a seamless and transparent integration of opportunistic resources into HEP computing at KIT.

KIT is currently further improving the resource management and job scheduling by taking into account I/O requirements of individual workflows as well as the available network bandwidth. In this talk an overview of the utilized technologies, the dynamic management and integration of resources as well as the status of the I/O-based resource and job scheduling is given.

Primary author

Matthias Jochen Schnepf (KIT - Karlsruhe Institute of Technology (DE))


Andreas Heiss (KIT - Karlsruhe Institute of Technology (DE)) Andreas Petzold (KIT - Karlsruhe Institute of Technology (DE)) Max Fischer (GSI - Helmholtzzentrum fur Schwerionenforschung GmbH (DE)) Eileen Kuhn (KIT - Karlsruhe Institute of Technology (DE)) Manuel Giffels (KIT - Karlsruhe Institute of Technology (DE)) Christoph Heidecker (KIT - Karlsruhe Institute of Technology (DE)) Gunter Quast (KIT - Karlsruhe Institute of Technology (DE))

Presentation materials