Indico has been upgraded to version 3.1. Details in the SSB
Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Setup and commissioning of a high-throughput analysis cluster

Nov 5, 2019, 3:30 PM
1h
Hall F (Adelaide Convention Centre)

Hall F

Adelaide Convention Centre

Poster Track 7 – Facilities, Clouds and Containers Posters

Speaker

Rene Caspart (KIT - Karlsruhe Institute of Technology (DE))

Description

Current and future end-user analyses and workflows in High Energy Physics demand the processing of growing amounts of data. This plays a major role when looking at the demands in the context of the High-Luminosity-LHC. In order to keep the processing time and turn-around cycles as low as possible analysis clusters optimized with respect to these demands can be used. Since hyperconverged servers offer a good combination of compute power and local storage, they form the ideal basis for these clusters.
In this contribution we report on the setup and commissioning of a dedicated analysis cluster setup at Karlsruhe Institute of Technology. This cluster was designed for use cases demanding high data-throughput. Based on hyperconverged servers this cluster offers 500 job slots and 1 PB of local storage. Combined with the 100 Gb network connection between the servers and a 200 Gb uplink to the Tier 1 storage, the cluster can sustain a data throughput of 1 PB per day.
In addition the local storage provided by the hyperconverged worker nodes can be used as cache space. This allows employing of caching approaches on the cluster, thereby enabling a more efficient usage of the disk space. In previous contributions this concept has been shown to lead to an expected speedup of 2 to 4 compared to conventional setups.

Consider for promotion No

Primary author

Rene Caspart (KIT - Karlsruhe Institute of Technology (DE))

Co-authors

Max Fischer (Karlsruhe Institute of Technology) Manuel Giffels (KIT - Karlsruhe Institute of Technology (DE)) Christoph Heidecker (KIT - Karlsruhe Institute of Technology (DE)) Eileen Kuhn (KIT - Karlsruhe Institute of Technology (DE)) Andreas Petzold (KIT - Karlsruhe Institute of Technology (DE)) Gunter Quast (KIT - Karlsruhe Institute of Technology (DE)) Andreas Heiss (KIT - Karlsruhe Institute of Technology (DE))

Presentation materials