Speaker
Benjamin Tovar Lopez
(University of Notre Dame)
Description
In this notebook talk we will demonstrate the Coffea Work Queue executor for analysis workflows. Work Queue is a framework for building large scale manager-worker applications. When used together with Coffea, it can measure the resources, such as cores and memory, that chunks of events need and adapt their allocations to maximize throughput. Further, we will demonstrate how the executor can dynamically modify the size of chunks of events when the memory available is not enough, and adapt it to desired resource usage. We will introduce its basic use for small local runs, and how it can automatically export the needed python environments when working in a cluster with no previous setup.
Authors
Benjamin Tovar Lopez
(University of Notre Dame)
Douglas Thain
