Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Predicting resource usage for enhanced job scheduling for opportunistic resources in HEP

Nov 7, 2019, 3:30 PM
1h
Hall F (Adelaide Convention Centre)

Hall F

Adelaide Convention Centre

Poster Track 7 – Facilities, Clouds and Containers Posters

Speaker

Max Fischer (Karlsruhe Institute of Technology)

Description

Job schedulers in high energy physics require accurate information about predicted resource consumption of a job to assign jobs to the most reasonable, available resources. For example, job schedulers evaluate information about the runtime, numbers of requested cores, or size of memory, and disk space. Users, therefore, specify those information when submitting their jobs and workflows. Yet, the information provided by users cannot be considered entirely accurate. There are several reasons for this, including the heterogeneity of resources, regular changes to the underlying workflows, external dependencies or even lack of knowledge. This inaccuracy can result in inefficient utilisation of assigned resources by either blocking unused resources or exceeding reserved resources.

With the increasing demand for the integration of opportunistic resources to extend the available WLCG computing resources, the accuracy of predicted resource consumption is of particular importance. Only an accurate prediction of resource consumption can enable a proper selection, allocation, and integration of resources to minimise the overall costs. We therefore propose to improve the indicated resource consumption of end-users with predictions to improve the resource utilisation of allocated opportunistic resources.
In this contribution, we present our results and the impact of our prediction for both, end-user workflows and production workflows including pilot jobs. Our work focuses on resource consumption of CPU and memory but presents a generic approach that is ready for future use of other resources, such as GPUs.

Consider for promotion No

Primary authors

Eileen Kuehn (Karlsruhe Institute of Technology) Max Fischer (Karlsruhe Institute of Technology)

Co-authors

Andreas Petzold (KIT - Karlsruhe Institute of Technology (DE)) Andreas Heiss (KIT - Karlsruhe Institute of Technology (DE))

Presentation materials