Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Real-time HEP analysis with funcX, a high-performance platform for function as a service

Nov 5, 2019, 11:00 AM
Riverbank R7 (Adelaide Convention Centre)

Riverbank R7

Adelaide Convention Centre

Oral Track 7 – Facilities, Clouds and Containers Track 7 – Facilities, Clouds and Containers


Dr Anna Elizabeth Woodard (University of Chicago)


The traditional HEP analysis model uses successive processing steps to reduce the initial dataset to a size that permits real-time analysis. This iterative approach requires significant CPU time and storage of large intermediate datasets and may take weeks or months to complete. Low-latency, query-based analysis strategies are being developed to enable real-time analysis of primary datasets by replacing conventional nested loops over objects with native operations on hierarchically nested, columnar data. Such queries are well-suited to distributed processing using a strategy called function as a service (FaaS).

In this presentation we introduce funcX---a high-performance FaaS platform that enables intuitive, flexible, efficient, and scalable remote function execution on existing infrastructure including clouds, clusters, and supercomputers. A funcX function explicitly defines a function body and dependencies required to execute the function. FuncX allows users, interacting via a REST API, to register and then execute such functions without regard for the physical resource location or scheduler architecture on which the function is executed---an approach we refer to as ``serverless supercomputing.'' We show how funcX can be used to parallelize a real-world HEP analysis operating on columnar data to aggregate histograms of analysis products of interest in real time. Subtasks representing partial histograms are dispatched as funcX requests with expected runtimes of less than a second. Finally, we demonstrate efficient execution of such analyses on heterogeneous resources, including leadership-class computing facilities.

Consider for promotion No

Primary authors

Yadu Babuji (University of Chicago) Dr Ben Blaiszik (University of Chicago) Dr Kyle Chard (University of Chicago) Dr Ryan Chard (Argonne National Laboratory) Prof. Ian Foster (University of Chicago) Dr Zhuozhao Li (University of Chicago) Tyler Skluzacek (University of Chicago) Dr Ana Trisovic (University of Chicago) Dr Anna Elizabeth Woodard (University of Chicago) Daniel S. Katz (University of Illinois)

Presentation materials