Sep 24 – 27, 2019
EC-JRC Ispra
Europe/Rome timezone

Planning for Future Scales and Complexity of HTCondor Pools in the CMS Experiment

Sep 25, 2019, 5:45 PM
35m
EC-JRC Ispra

EC-JRC Ispra

European Commission – Joint Research Centre Via Enrico Fermi, 2749 I - 21027 Ispra (VA) Italy N45° 48' 36.09'' E008° 37' 16.72'' N45° 48.601 E008° 37.278 45.80998, 8.62135 https://www.openstreetmap.org/#map=17/45.80998/8.62135
HTCondor presentations and tutorials Workshop presentations

Speaker

James Letts (Univ. of California San Diego (US))

Description

The resource needs of high energy physics experiments such as CMS at the LHC are expected to continue to grow significantly over the next decade, and will be more and more satisfied by computing capacity with non-standard characteristics. This presents challenges not only of scale but of complexity in resource provisioning and allocation. In this contribution, we will present results of recent HTCondor scale tests we have conducted using the CMS Global Pool Integration Test Bed (ITB) employing the multi-threaded Negotiator, where we have pushed the size of the pool to the maximum limits with currently-available hardware and explored effective performance limitations of the submit nodes in our infrastructure with realistic payloads. We will also discuss recent integration of resource-specific job matching conditions to satisfy HPC and Cloud use cases, where resources may not be suitable for running all kinds of workflows. Finally, we will review some specific use cases that we have difficulty solving with the current implementation of HTCondor.

Desired slot length 20 minutes, unless you want to give us more.
Speaker release Yes

Primary author

James Letts (Univ. of California San Diego (US))

Co-authors

Antonio Perez-Calero Yzquierdo (Centro de Investigaciones Energéti cas Medioambientales y Tecno) Saqib Haleem (National Centre for Physics (PK))

Presentation materials