Using Spark for Physics

Wednesday May 4, 2022, 4:30 PM → 5:30 PM Europe/Zurich

Vidyo only

Gordon Watts (University of Washington (US))

• https://github.com/LucaCanali/Miscellaneous/tree/master/Spark_Physics

Using modern Spark as an analysis tool

4:30 PM → 5:30 PM Investigating Apache Spark for Physics Analysis

Apache Spark is a very successful open-source tool for data processing, over the last few years Spark and platforms built around it have seen large adoption in industry. This talk will focus on the use of Spark and its DataFrame API in the context of HEP. We will go through a few demos of some simple analyses implemented on Jupyter notebooks using Apache Spark APIs. We will also briefly review some related work on Spark DataFrames for large scale Physics data preparation/reduction. Based on those experiences we will discuss the key features of Spark and its ecosystem that can be useful for Physics analysis, and what still needs improvement, compared to the current state of the art analysis software.

Speaker: Luca Canali (CERN)

ApacheSpark_for_Physics_May2022.pdf

ApacheSpark_for_Physics_May2022.pptx