Sep 21 – 25, 2020
(teleconference only)
Europe/Paris timezone

Lightweight Site-Specific Dask Integration for HTCondor at CHTC

Sep 25, 2020, 3:45 PM

HTCondor user presentations Workshop session


Mr Matyas Selmeci (University of Wisconsin - Madison)


Dask is an increasingly-popular tool for both low-level and high-level parallelism in the Scientific Python ecosystem. I will discuss efforts at the Center for High Throughput Computing at UW-Madison to enable users to run Dask-based work on our HTCondor pool. In particular, we have developed a "wrapper package" based on existing work in the Dask ecosystem that lets Dask spawn workers in the CHTC pool without users needing to be aware of the infrastructure constraints we are operating under. We believe this approach is useful as a lightweight alternative to dedicated, bespoke infrastructure like Dask Gateway.

Speaker release Yes

Primary author

Mr Matyas Selmeci (University of Wisconsin - Madison)


Brian Paul Bockelman (University of Wisconsin Madison (US))

Presentation materials