PyHEP 2024 - "Python in HEP" Users Workshop (online)

1 Jul 2024, 14:00 → 4 Jul 2024, 23:00 Europe/Zurich

Eduardo Rodrigues (University of Liverpool (GB)), Graeme A Stewart (CERN), Jim Pivarski (Princeton University), Matthew Feickert (University of Wisconsin Madison (US)), Nikolai Hartmann (Ludwig Maximilians Universitat (DE))

The PyHEP workshops are a series of workshops initiated and supported by the HEP Software Foundation (HSF) with the aim to provide an environment to discuss and promote the usage of Python in the HEP community at large. Further information is given on the PyHEP Working Group website.

PyHEP 2024 will be an online workshop. It will be a forum for the participants and the community at large to discuss developments of Python packages and tools, exchange experiences, and inform the future evolution of community activities. There will be ample time for discussion.

The agenda is composed of plenary sessions:

1) Topical sessions with 3 types of presentations (see the call for abstracts).

2) Presentations following up from topics discussed at PyHEP 2023 and PyHEP.dev 2023.

 

Registration will open on March 12th. There are no workshop fees.

 

You are encouraged to register to the PyHEP WG Gitter channel and/or to the HSF forum to receive further information concerning the organisation of the workshop. Workshop updates and information will also be shared on the workshop Twitter in addition to email. Follow the workshop @PyHEPConf.

SSL BinderHub link: binderhub.ssl-hep.org

Organising Committee

Eduardo Rodrigues - University of Liverpool (Chair)                         
Graeme A. Stewart - CERN                         
Jim Pivarski - Princeton University                         
Matthew Feickert - University of Wisconsin-Madison                         
Nikolai Hartmann - Ludwig-Maximilians-Universität Munich                         

 

Sponsors

The event is kindly sponsored by

 

Monday, 1 July

14:00 → 16:00 Plenary Session Monday

Conveners: Eduardo Rodrigues (University of Liverpool (GB)), Nikolai Hartmann (Ludwig Maximilians Universitat (DE))

14:00 Welcome and workshop overview

Speaker: Eduardo Rodrigues (University of Liverpool (GB))

EduardoRodrigues_2024-07-01_PyHEP2024_Intro.pdf

14:10 Plothist: visualize and compare data in a scalable way and a beautiful style.

Plothist is a Python package that provides easy-to-use functions for visualizing and comparing histograms generated with boost-histogram. The package provides an out-of-the-box, publication-ready default style that significantly reduces the effort required to produce publishable-quality figures, while retaining all the flexibility offered by matplotlib. Specifically, plothist includes functions for comparing data with models consisting of any number of stacked and unstacked components that are either histograms or functions, using various methods such as ratio, pull, and difference, with automatic uncertainty propagation. In addition, the package includes a variable manager that allows to store and manage plotting information for a large number of variables, streamlining the modification process through a user-friendly YAML file. Finally, plothist comes with comprehensive and easy-to-navigate documentation and an example gallery, making it accessible to users of all skill levels.

Speakers: Cyrille Praz, Tristan Fillinger (KEK / IPNS)

documentation

github

live_demo (binder)

slides_plothist.pdf

14:40 VLQcalc: a Python Module for Calculating Vector-like Quark Couplings

We present a python module named VLQcalc which can be used for employing coupling parameters of vector-like quarks (VLQs) within model framework. It facilitates the conversion of couplings between different parametrization of VLQ models and computes these couplings depending on the width-to-mass ratio ($\Gamma/m$), with given branchings. By utilizing a modified version of Genetic Algorithm (mGA) for coupling constant computations it provides parameter-dependent results. This innovative algorithm allows for swift and precise determination of correlated coupling constants with choosen branching ratios for different representations. In addition, VLQcalc can generate input cards for MadGraph5 (MG5) for event generation. The module's capability to efficiently manage the process from calculating coupling constants to prepare MG5 input card will accelerate the tasks in the new studies.

Speaker: Ali Can Canbay (Ankara University (TR))

GitHub Repository

Run on Binder (mybinder)

Run on Binder (ssl-hep)

VLQcalc_PyHEP24.pdf

14:50 The two flavors of Python 3.13

This is a talk in two parts. The first part goes over what’s new in Python 3.13, the most forward-thinking release we’ve ever had. For the first time in 30+ years, Python has a new REPL implemented in Python with exciting new features. Python exceptions have color by default now, and several errors have been improved. There are also new typing features along with a new deprecated decorator that can be used by typing and at runtime. Lots of other smaller features were added, as well. And there now is a (disabled by default) JIT in CPython!

The second part of the talk will focus on a single feature that requires a second copy of Python: “free-threaded” Python, which for the first time ever removes the GIL and enables true multithreading. We’ll very briefly look at how it was done, and we will look at how to use it both from pure Python code as well as with a compiled extension, using scikit-build-core. This grand experiment might be the largest change to CPython we’ve ever had if it succeeds, and its success depends on all of you preparing for true multi-threaded code in Python.

Speaker: Henry Fredrick Schreiner (Princeton University)

Talk repo

thetwoflavorsofpython313.pdf

15:20 The new Python client library of ServiceX, the novel data delivery system

Effective data extraction has been one of major challenges in physics analysis and will be more important in the High-Luminosity LHC era. ServiceX provides a novel data access and delivery by exploiting industry-driven software and recent high-energy physics software in the python ecosystem. In this talk, the newly designed client library will be extensively introduced with various practical examples. ServiceX in a physics analysis pipeline will be briefly discussed with an example. The future of ServiceX also will be briefly described.

Speaker: Kyungeon Choi (University of Texas at Austin (US))

ServiceX_Choi

16:00 → 16:30 BREAK

16:30 → 18:00 Plenary Session Monday

Conveners: Graeme A Stewart (CERN), Matthew Feickert (University of Wisconsin Madison (US))

16:30 b2luigi — bringing batch 2 luigi!

Workflow managers help structure the code of pipelined jobs by defining and managing dependencies between tasks in a clear and easy-to-understand fashion. This abstraction allows independent tasks to be automatically parallelised more independently of computing systems. Additionally, workflow managers help keep track of different tasks’ outputs and inputs.

b2luigi is an extension of the workflow manager luigi and offers easy integration with batch systems such as HTCondor and LSF, allowing the combination of different systems within one workflow.

b2luigi also provides additional interfaces tailored for Belle II workflows, allowing smooth interaction with the Belle II analysis software framework and distributed computing. Workflows such as VIBE, an automated Monte Carlo validation framework, the Systematics Framework, and many Belle II physics analyses have been automated using b2luigi.

As the current maintainers of b2luigi and Belle II users, we look forward to discussing our experiences and plans for this tool at the PyHep 2024 conference.

Speakers: Alexander Heidelbach, Jonas Eppelt (Karlsruher Insititute of Technology (KIT))

B2luigi_PyHEP_01072024.pdf

Documentation

GitHub

17:00 The New PDG Python API

The Python package pdg provides a high-level tool for programmatically accessing the data published by the Particle Data Group (PDG) in the Review of Particle Physics. We will give an overview of the package and the functionality it provides, and will then demonstrate its use and show a number of examples.

Speakers: Juerg Beringer (Lawrence Berkeley National Laboratory (LBNL)), Matt Kramer (Lawrence Berkeley National Laboratory)

1-PDG-Introduction.pdf

Binder (mybinder)

Binder (ssl-hep)

Evaluated notebook

Tuesday, 2 July

14:00 → 16:00 Plenary Session Tuesday

Conveners: Graeme A Stewart (CERN), Nikolai Hartmann (Ludwig Maximilians Universitat (DE))

14:00 Multi-scale cross-attention transformer encoder for event classification

In this presentation, I will emphasize the significance of employing attention-based transformer models for analyzing particle clouds. Specifically, I will delve into the utilization of a multi-modal transformer model equipped with both self-attention and cross-attention mechanisms to effectively analyze various scales of inputs. These inputs include the intricate local substructures of jets as well as the broader, high-level reconstructed kinematics. Additionally, I will introduce interpretation techniques such as attention maps and Grad-CAM to provide insights into the network's outcomes.

The network structure based on arXiv:2401.00452 and the public code is given in https://github.com/AHamamd150/Multi-Scale-Transformer-encoder

Speaker: Dr Ahmed Hammad (KEK)

14:30 Reading RNTuple data with Uproot

RNTuple is an upcoming columnar data storage format with a variety of improvements over TTree. It is available as an experimental feature in the latest versions of ROOT, and a stable version is expected in the not-so-distant future. The Uproot Python library has kept up changes in the RNTuple specification and currently supports reading most RNTuple files. In this talk, I will briefly introduce the RNTuple format and its benefits, demonstrate how to use the Uproot library to read RNTuple data, and discuss current capabilities, limitations, and future work that will be done to support as much of the RNTuple specification as possible.

Speaker: Andres Rios-Tascon (Princeton University)

Slides

Talk repo

15:00 Distributed Columnar HEP analysis using coffea + dask

This talk will explore the recent advancements in the Coffea framework, a set of tools and wrappers designed to facilitate columnar Collider High-Energy Physics (HEP) analyses with user-friendly syntax. With the release of AwkwardArray 2.0, the Dask parallel processing library is now a core part of HEP analysis, and provides powerful computing abstractions through the task graphs it produces.

Coffea has been retooled to harness this new infrastructure. By integrating uproot, dask-awkward, and dask-histogram, analyses written using Coffea can be automatically optimized for data transport and distributed across clusters.

This talk will demonstrate the current capabilities of dask-awkward, dask-histogram, and Coffea through an extended Jupyter-based tutorial. Attendees will see practical examples showcasing the functionality of these tools and learn how to use them to develop their own analyses. The basics of task-graph building and lazy evaluation will be covered, along with tips and caveats for migrating analysis code from Coffea 0.7 to this enhanced toolset.

Speaker: Iason Krommydas (Rice University (US))

GitHub Repository

Run on BinderHub

Run on MyBinder

16:00 → 16:30 BREAK

16:30 → 18:00 Plenary Session Tuesday

Conveners: Eduardo Rodrigues (University of Liverpool (GB)), Matthew Feickert (University of Wisconsin Madison (US))

16:30 general model fitting with zfit and hepstats

Statistical inference using likelihood based methods is a key step in most HEP analyses. This talk will introduced general likelihood fitting using zfit for model building and minimization as well as hepstats for interval estimations and targets a wide audience.
The talk will be based on notebooks and start from simple unbinned and binned fits using examples that incorporate other Python and Scikit-HEP libraries to showcase the usage in an analysis flow.
It continues to more elaborate topics covering simultaneous fits, multidimensional PDFs and custom PDFs and statistical methods such as limit setting.

Speaker: Jonas Eschle (Syracuse University (US))

17:30 Recent developments and user contributions in zfit

In this talk, we will explore the newest developments and user contributions of zfit, a cutting-edge Python library designed for fitting binned and unbinned likelihoods within the High Energy Physics (HEP) analysis ecosystem. Built on TensorFlow, zfit offers a high-level interface for defining and fitting models, facilitating efficient and robust analysis workflows.

Over the past five years, zfit has evolved significantly, achieving a stable core and comprehensive feature set. Recent updates have increased the compatibility with the Python ecosystem for binned and unbinned fits and greatly improved the user-friendliness through various features. zfit-physics, an extension to zfit that provides physics specific models, has been largely extended thanks to a low-entry barrier for user contributions. We will show-case some of the new PDFs and outline how the community can create their own PDFs and make them available to a wide audience by contributing to zfit -- thereby shaping zfit's future trajectory.

Speaker: Iason Krommydas (Rice University (US))

GitHub Repository

Run on BinderHub

Run on MyBinder

Wednesday, 3 July

14:00 → 16:00 Plenary Session Wednesday

Conveners: Eduardo Rodrigues (University of Liverpool (GB)), Jim Pivarski (Princeton University)

14:00 Quantum Machine Learning in High Energy Physics with Qibo

Over the past three decades, Quantum Computing (QC) has emerged as a prominent field of research, with the intent of exploring whether and in which context it can help to expediently address problems that are either challenging or infeasible to solve using classical methods. In particular, High-Energy Physics (HEP) has been recently identified as a promising playground to challenge QC routines.
Alongside with this research, the development and maintenance of robust libraries are essential, enabling users to seamlessly implement applications and interface with QC routines.
We introduce Qibo, a comprehensive and open-source framework designed for quantum computing. Qibo provides an extensive range of modules for the simulation, control, and calibration of quantum devices, which can be accessed through a simple High-Level API in Python. Thanks to its modularity, Qibo allows effortlessly execution of its high-level implementation onto any type of hardware accelerator: multi-threading CPU, GPU and multi-GPU for quantum simulation on classical hardware (using state-vector and tensor network approaches) and Quantum Processing Units (QPU) for execution on self-hosted quantum devices.
It also includes a suite of application packages, notably a module dedicated to developing and training Quantum Machine Learning (QML) models. This module facilitates easy integration with popular machine learning frameworks such as TensorFlow and PyTorch.
After a concise overview of the project goals and introducing some of the Qibo primitives, we highlight our specialized models for High-Energy Physics applications. In particular, we describe and train a QML model designed to fit the proton parton distribution functions.

Speaker: Matteo Robbiati (Università degli Studi e INFN Milano (IT))

15:00 RootInteractive tool for multidimensional statistical analysis, machine learning and analytical model validation

RootInteractive is a general purpose tool for multidimensional statistical analyses, mainly used in the ALICE experiment at CERN. This Python-based tool enables dynamic, interactive visualisation and data aggregation and enhances capabilities on both the server and client side, expanding analysis possibilities for researchers and educators. As machine learning (ML) becomes increasingly important in multidimensional data analysis, interpreting ML models and assessing their uncertainties proves challenging, especially when the analyses reduce dimensionality, which can lead to misleading conclusions.

Our goal with RootInteractive is to streamline the management of complex multidimensional challenges. It allows users to visualise and fit multidimensional functions, incorporate uncertainty and bias, validate assumptions and define the functional composition of parametric and non-parametric analytical functions. This includes the use of symmetries and the development of multidimensional “invariant” functions/alarms, which are crucial for the validation of machine learning models and the optimisation of parameters in reconstruction, calibration and simulation.

RootInteractive uses a declarative programming paradigm that enables the construction of programme structures and the expression of computational logic without detailed control flow, making it easily accessible to professionals, students and educators alike. The tool supports interactive visualisation for both unbinned and binned data, facilitates n-dimensional histogramming/projection and enables the extraction of derived aggregated information on both the server (Python/C++) and client (JavaScript) side. It is compatible with client/server configurations via Jupyter and can also be used as a standalone client application or dashboard.

RootInteractive uses both lossy and lossless data compression and enables interactive analysis of large data sets — up to about O(10^7) entries times O(25) attributes — in a compressed browser environment of about 500 MB. By applying representative downsampling (O(10^-2 to 10^-3)) followed by reweighting or pre-aggregation on servers or batch farms, it facilitates effective multidimensional analysis of monthly/annual ALICE statistics for calibration, reconstruction validation, QA/QC or statistical/physical analyses.

Recent development in RootInteractive have focussed on better integration of downsampling of representative data and support for conditional reweighting in interactive multidimensional aggregation. An important development was the integration of a domain-specific language that simplifies integration with RDataFrame and thus facilitates complex data operations.

Speaker: Marian I Ivanov (GSI - Helmholtzzentrum fur Schwerionenforschung GmbH (DE))

15:30 AwkwardArrays in Julia for High-Energy Physics Data Analysis

AwkwardArrays are well known to Python users for their powerful capabilities in handling irregular, nested data structures with ease. While Python has been the primary language for implementing AwkwardArrays, the recent integration into Julia offers new possibilities for data scientists and researchers.
In this talk, we will explore the implementation and advantages of using AwkwardArrays within the Julia programming environment, tailored for an audience familiar with Python. We will begin with an overview of AwkwardArrays, highlighting their key features and importance in HEP data analysis. Following this, we will look into the specifics of Julia’s implementation, showing how it leverages Julia’s strengths such as just-in-time compilation and multiple dispatch to achieve superior performance.
Through a series of live demonstrations and code examples, participants will learn how to create, manipulate, and analyze AwkwardArrays in Julia. Key topics will include efficient array creation, advanced indexing techniques, broadcasting, and aggregation. Additionally, we will discuss interoperability between Python and Julia, providing practical guidance on how to integrate Julia-based AwkwardArrays into existing Python workflows for a more efficient and scalable analysis.

Speaker: Ianna Osborne (Princeton University)

16:00 → 16:30 BREAK

16:30 → 18:00 Plenary Session Wednesday

Conveners: Eduardo Rodrigues (University of Liverpool (GB)), Graeme A Stewart (CERN)

16:30 Easy Columnar File Conversion with 'hepconvert'

Though columnar file formats are popular among HEP users, the process to convert between file formats has multiple steps, and generally requires the use of one I/O package per file format. Often users need to customize the process as well, either due to memory constraints or to modify the data before writing it to a new file. This entails both more lines of code and experience with I/O packages, and in some cases knowledge about each data format.
To streamline this process and save user’s time, we are developing the Python package ‘hepconvert.’ This package aims to simplify columnar file conversions and common customizations down to single function between file formats Parquet, ROOT, and HDF5 files. It uses pre-existing functions from reputable columnar I/O packages such as Uproot, Awkward Array, and h5py, with additional builtin features for common customizations. The customizations are added at user request and include automatic reading and writing in batches, compression setting, branch skimming and slimming, histogram summing, and more. In addition to making the features in hepconvert, we are also adding relevant functionality to Uproot that will eventually be included in hepconvert; adding new TBranches to existing TTrees.

Speaker: Zoë Bilodeau (Princeton University (US))

17:00 A new SymPy backend for vector: uniting experimental and theoretical physicists

Vector is a Python library for 2D, 3D, and Lorentz vectors, especially arrays of vectors, to solve common physics problems in a NumPy-like way. Vector currently supports creating pure Python Object, NumPy arrays, and Awkward arrays of vectors. The Object and Awkward backends are implemented in Numba to leverage JIT-compiled vector calculations. Furthermore, vector also supports JAX and Dask operations on Awkward arrays of vectors.

We introduce a new SymPy backend in vector to allow symbolic computations on high energy physics vectors. Along with experimental physicists using vector for numerical computations, the SymPy backend will enable theoretical physicists to utilize the library for symbolic computations. Since the SymPy vector classes and their momentum equivalents operate on SymPy expressions, all of the standard SymPy methods and functions work on the vectors, vector coordinates, and the results of operations carried out on vectors. Moreover, vector’s SymPy backend will create a stronger connection between software used by experimentalists and software used by theorists.

This talk will introduce vector and its backends to the users and funnel down to the SymPy backend. Finally, vector’s SymPy backend is relatively new; hence, we aim to collect suggestions and recommendations from both theoretical and experimental physicists.

Speaker: Saransh Chopra (Princeton University (US))

Thursday, 4 July

14:00 → 16:00 Plenary Session Thursday

Conveners: Eduardo Rodrigues (University of Liverpool (GB)), Jim Pivarski (Princeton University)

14:00 Metaheuristic optimization for artificial neural networks and deep learning architectures

Classical minimization methods, like the steepest descent or quasi-Newton techniques, have been proved to struggle in dealing with optimization problems with a high-dimensional search space or subject to complex nonlinear constraints, in addition to requiring continuous cost functions. For these reasons, in the last decade, the interest on metaheuristic nature-inspired algorithms has been growing steadily, due to their flexibility and effectiveness.

In this talk I will present a new python package which implements several metaheuristic algorithms (MHAs) for optimization problems. Unlike other available tools, it can be used not only for unconstrained problems, but also for problems subjected to inequality constraints and for integer or mixed-integer problems. Within the HEP community, a particularly interesting use case of MHAs is for the optimization of artificial neural networks (ANNs) and deep learning (DL) architectures, in terms of weights, hyper-parameters, and so on. This is indeed known to be one of the most challenging problems in machine learning and traditional gradient-based learning algorithms sometimes suffer from stucking at local minima and from the need of continuous cost functions. For these reasons, several authors have investigated the use of MHAs, even by combining some of them (hybrid MAHs), in optimizing the parameters involved in the training process of ANNs and, more recently, of DLs. In this talk I will also give an overview on this subject and show how to use the new package to integrate MHAs with ANN and DL architectures.

Speaker: Prof. Davide Pagano (Universita di Brescia (IT))

14:30 Constructing model-agnostic likelihoods, a method for the reinterpretation of particle physics results

Experimental High Energy Physics has entered an era of precision measurements. However, measurements of many of the accessible processes assume that the final states' underlying kinematic distribution is the same as the Standard Model prediction. This assumption introduces an implicit model-dependency into the measurement, rendering the reinterpretation of the experimental analysis complicated without reanalysing the underlying data.

We present a novel reweighting method in order to perform reinterpretation of particle physics measurements. It makes use of reweighting the Standard Model templates according to kinematic signal distributions of alternative theoretical models, prior to performing the statistical analysis. The generality of this method allows us to perform statistical inference in the space of theoretical parameters, assuming different kinematic distributions, according to a beyond Standard Model prediction.

The implementation - redist - is an extension to the pyhf software. It can easily be interfaced with the EOS software, which allows us to perform flavor physics phenomenology studies. Beyond the pyhf or HistFactory likelihood specification, only minimal information is necessary to make a likelihood model-agnostic and hence easily reinterpretable. We showcase that publishing such likelihoods is crucial for a full exploitation of experimental results.

Speaker: Lorenz Gärtner (LMU)

15:00 Checkpointing for long running Machine Learning Tasks

The increasing application of Machine Learning (ML) in High Energy Physics (HEP) analysis and reconstruction necessitates the use of GPUs. The extensive runtimes associated with training neural networks make them vulnerable to runtime constraints and failures.

Checkpointing, which involves storing the current state of the training persistently, offers a solution to these challenges. It allows for continuing training at a later time or different location, providing resilience against failures and adherence to time constraints.

Moreover, checkpointing contributes to sustainable computing efforts. For instance, training can be scheduled during periods of abundant renewable energy supply and paused when the supply is limited.

This presentation introduces a Python interface that consolidates common HEP community tools for storing checkpoints and rescheduling tasks. Examples on how to use this tool in combination with different setups like luigi workflow management or htcondor will be shown.

Speaker: Jonas Eppelt (Karlsruher Insititute of Technology (KIT))

15:30 Workshop close-out

Speaker: Eduardo Rodrigues (University of Liverpool (GB))