Speaker
Mr
Matyas Selmeci
(University of Wisconsin - Madison)
Description
Dask is an increasingly-popular tool for both low-level and high-level parallelism in the Scientific Python ecosystem. I will discuss efforts at the Center for High Throughput Computing at UW-Madison to enable users to run Dask-based work on our HTCondor pool. In particular, we have developed a "wrapper package" based on existing work in the Dask ecosystem that lets Dask spawn workers in the CHTC pool without users needing to be aware of the infrastructure constraints we are operating under. We believe this approach is useful as a lightweight alternative to dedicated, bespoke infrastructure like Dask Gateway.
| Speaker release | Yes |
|---|
Primary author
Mr
Matyas Selmeci
(University of Wisconsin - Madison)
Co-author
Brian Paul Bockelman
(University of Wisconsin Madison (US))
