Stefan Karpinski - State of Julia

• Stefan Karpinski

Is Julia ready to be adopted by HEP?

• Tamas Gal (University of Erlangen / Erlangen Centre for Astroparticle Physics)

The Julia programming language was created 10 years ago and is now a mature and stable language with a large ecosystem including more than 80,000 third-party packages. It was designed for scientific programming to be a high-level and dynamic language as Python is, while achieving runtime performances comparable to C/C++ or even faster. With this, we ask ourselves if the Julia language and its ecosystem is ready now for its adoption by the High Energy Physics community. We will report on a number of investigations and studies of the Julia language that have been done for various representative HEP applications, ranging from computing intensive initial data processing of experimental data and simulation, to final interactive data analysis and plotting. Aspects of collaborative code development of large software within a HEP experiment has also been investigated: scalability with large development teams, continuous integration and code test, code reuse, language interoperability to enable a adiabatic migration of packages and tools, software installation and distribution, training of the community, benefit from development from industry and academia from other fields.

Engaging the HEP community in Julia

• Jim Pivarski (Princeton University)

Julia's *technical* qualifications as a data analysis language for HEP are obvious: it combines the interactivity of dynamic scripting with the speed of statically typed compilation. It's well-posed to replace Python and C++ in HEP, at least from the stage of analysis objects (e.g. NanoAOD in CMS and PHYSLITE in ATLAS) to final publication. However, developers don't select a language purely on technical merits. Using the same language as one's peers has several advantages: sharing code, being able to talk about code in the same ways, and having access to the same libraries and tools. Also, programming language communities develop personalities—shared priorities about performance, readability, future-proofing, tooling, etc. Choosing a language based on popularity is a self-reinforcing proposition, though. It biases against new languages, even when they have qualitatively new and technologically better features. Changing the HEP analysis community from its current mixture of Python and C++ to Julia will be like moving from one local minimum to another, more optimal, minimum. There is a barrier to cross and we will need to find the best pathway over it, as well as short-term, local motivations—not just the technical merits of a 100% Julia future—to convince members of the community that each step is worth the effort. In this talk, I'll discuss the general challenge in terms of what I consider to be a specific and partial solution—sharing columnar data between Python and Julia using AwkwardArray.jl. I'll present some of the differences between the Python perspective and the Julia perspective on technical issues like type-granularity, indexing, object orientation/syntax/naming, and packaging, but also community values, such as those expressed in the [Zen of Python][1] (now 20 years old) and the [Greed of Julia][2] (12 years old), how those values have changed over time, and what may need to change for widespread adoption in HEP. [1]: https://peps.python.org/pep-0020/ [2]: https://julialang.org/blog/2012/02/why-we-created-julia/

Maintaining Large Scale Julia Ecosystems

• Chris Rackauckas

Reproducible Science: Why it matters and how to achieve it...

• George Datseris

TBA

Unit and Integration testing in modularized julia package eco-systems

• Simeon Ehrig

In high-energy physics, we want to simulate complex physical processes that require computational resources of the fastest HPC systems in the world. In order to fully use the computational resources, software that is maintainable, performant and extensible is required. To achieve these goals, automated testing is essential. Modern Julia HEP software is modularized into sub-packages to improve maintainability and extensibility. This creates new challenges for automated testing. Unit tests and integration tests are required. If the code of one package is changed, unit tests ensure that the package is still working, whereas integration tests ensure that in dependent packages no functionality break is caused by the change. Using the QED.jl [1] project as an example, I will demonstrate how we implemented unit and integration tests for the main QED.jl package and its sub-packages. [1] https://github.com/QEDjl-project/QED.jl

BinaryBuilder.jl: distributing binary libraries for Julia packages

• Mose Giordano

The Julia programming languages promises to solve the so-called "two-language problem", so that users don't need to rewrite code in other languages to achieve better performance. However, we don't live in a Julia-only world, and many high-quality libraries that we don't want to miss out have been already written in other languages. `BinaryBuilder.jl` allows you to take software written in compiled languages such as C, C++, Fortran, Go or Rust, and build precompiled binaries for a plethora of different platforms that can be used from Julia packages, but not only. Automatically generated thin wrappers, called JJLs, allow users to seamlessly install binaries just like regular Julia packages, and the other Julia packages can depend on JLLs. Additionally, Julia's package manger records JLLs in the manifest file capturing the content of an environment for reproducibility, so that not only does it record Julia packages, but also binary libraries in other languages. In this talk we will describe the architecture of `BinaryBuilder.jl`, and how it piggybacks Julia's package manager and the Artifacts system to safely deliver the binaries to the end users.

Automatic generation of Julia bindings to libraries written C++

• Philippe Gras (Université Paris-Saclay (FR))

HEP has a large legacy of code written in C++. Access to this code from Julia can be done using the CxxWrap.jl package. It requires to write some piece of C++ code that specifies the classes and methods to bind. Despite its simplicity, writing this code and maintaining it can be time-consuming for large libraries. We will present the WrapIt! tool that automatizes the generation of the code. The tool has been designed to allow adding Julia binding to a C++ library with minimal effort and code maintenance. The list of methods and classes to bind are extracted from the library header file and the configuration required by the tool is minimal. We will present how the automation is implemented, how the tool is used and its current limitations. The development to foresee to remove these limitations will be also presented.

UnROOT.jl update: RNTuple, PHYSLITE, and future priorities

• Jerry 🦑 Ling (Harvard University (US))

We present updates on the cornerstone of JuliaHEP analysis that enables users to work with data sets in `.root` files efficiently. The update focuses on the latest development in ROOT community: RNTuple, which is set up replace TTree in Run 4. We demonstrate a high level of compatibility even at an early stage, and demonstrate Julia design pattern involved. Finally, we present a potential application in Run 3 in ATLAS collaboration, where a new collaboration-wide format PHYSLITE can already be ingested by UnROOT.jl. We would also like to open a discussion about the future priorities and more experiment-specific needs from UnROOT.jl for wider adoption.

KM3io.jl - An example how to wrap UnROOT to make ROOT files more accessible

• Tamas Gal (University of Erlangen / Erlangen Centre for Astroparticle Physics)

Handling ROOT files with pure Julia has been made possible by the UnROOT.jl package. It can however be complicated for a user if an experiment is heavily using custom classes and complicated branch structures. KM3io.jl was created to deal with the ROOT files produced by the KM3NeT DAQ, MC and reconstruction workflows. This talk shows how UnROOT.jl and low-level access can be hidden from the user, so that it feels like handling the ROOT file in its original (C++) framework.

Corpuscles.jl - A package to access particle properties from the PDG particle catalogue

• Johannes Schumann

The availability of physics packages in the Julia ecosystem is essential for model building and setting up analyses without reinventing the wheel at every corner. In the HEP environment the properties of elementary and sub-atomic particles are a widely needed information. Thus, the package Corpuscles.jl was developed based on the official PDG particle catalogue.

PDGdb.jl Particle Database wrangling

• Mikhail Mikhasenko (Excellence Cluster ORIGINS)

**PDGdb.jl** is a Julia module designed for seamless interaction with the Particle Data Group (PDG) SQLite database. This package presents a straightforward interface, enabling users to access and query the comprehensive particle physics data stored in the public PDG database, including their properties and decay modes recorded in the table. The public PDG database, a compilation of five databases in SQLite format, is inherently intricate. This complexity arises from its vastness and the historical accumulation of data contributed by hundreds' encoders worldwide over several decades. A notable challenge is the presence of hadronic resonances within isospin multiplets. These resonances, while sharing most properties, are assigned the same PDG identification number (pdgid). PDGdb.jl stands out as a collection of tailored data-frame tools, facilitating efficient data manipulation and exploration. The package provides utilities for particle search, retrieval of particle properties, and the extraction of specific information. When compared to other related packages, such as the Particles from scikit and Corpuscles.jl, PDGdb.jl offers data in its full complexity. Users can access all recordings related to measurements on particle-decay modes. While the PDG Python API interacts with the same database, it does so in a more sophisticated manner, focusing on reliable information but potentially limiting exploratory aspects.

A 3D Event Display based on Makie for Cherenkov Neutrino Detectors

• Tamas Gal (University of Erlangen / Erlangen Centre for Astroparticle Physics)

RainbowAlga.jl is a tiny package which utilises the Makie ecosystem to create an interactive 3D visualisation tool for displaying neutrino events in KM3NeT detectors. This talk introduces the package and shows how easy it is to create an easy-to-deploy cross-platform application with an interactive graphical user interface. The software can be tweaked to display events from similar Cherenkov neutrino detectors like e.g. IceCube, Baikal BDUNT /GVDl, P-One or Trident.

BAT.jl, the Bayesian analysis toolkit in Julia

• Oliver Schulz (Max Planck Society (DE))

The Bayesian Analysis Toolkit (BAT.jl) is an open source software package that provides user-friendly tooling to tackle statistical problems encountered in Bayesian inference. BAT.jl is focused on use cases that cannot easily be expressed in a domain-specific language, but instead require application-specific custom-written models. Typical examples are use cases that need to access large data sets or depend on external software frameworks, or that are computationally very costly and need to be manually optimized. For example, BAT.jl was recently used to infer parton PDFs from HERA data by interfacing inference algorithms and prior definitions in Julia with a forward-model based on the QCDNUM library in Fortran. BAT.jl has also been used to infer Proton bunch parameters in the AWAKE experiment. In combination with DifferentialEquations.jl, BAT.jl has been used to infer parameters of compartmental epidemiological models. For the standard task in Bayesian inference, sampling from the posterior, BAT.jl supports a variety sampling algorithms like MCMC, nested sampling and importance sampling. An option to perform variational inference based on machine-learned normalizing flowsis currently in preparation, as well as deeper integration with the Julia measure theory packages. BAT.jl can compute Bayesian evidences by integrating over the the non-normalized posterior using our adaptive harmonic mean algorithm (AHMI) algorithm, bridge-sampling, or several common integration algorithms like VEGAS. To speed up MCMC sampling and improve sampling quality, BAT supports parallelized sampling via space partitioning. BAT.jl's capability to automatically transform prior distributions to the space (normal or uniform) preferred for a given algorithm can also be utilized to perform Bayesian guided maximum likelihood estimation, which makes ML fits converge faster while still de-facto independent of prior choices. In addition to presenting BAT we offer a tutorial so attendees can quicky and easily try out BAT.jl for themselves.

EFTfitter.jl - A tool for combining measurements (not only for EFTs)

• Cornelius Grunwald (Technische Universitaet Dortmund (DE))

Effective field theories (EFTs) gained a lot of popularity in the HEP community in recent years, as they provide a powerful approach to indirect searches for heavy physics beyond the Standard Model at current collider energies. Constraining the free parameters of EFTs is usually only possible by combining measurements of multiple observables. However, combining measurements on likelihood-level is a non-trivial task and typically not feasible outside the respective experimental collaborations. The EFTfitter.jl package offers a user-friendly interface and an easy-to-interpret statistical approach to combine (possibly correlated) measurements of different observables by constructing a common likelihood assuming Gaussian measurements. By interfacing to the Bayesian Analysis Toolkit (BAT.jl) it is possible to perform Bayesian inference to constrain underlying model parameters, such as the Wilson coefficients of EFTs. Although the methods used in EFTfitter.jl are particularly well suited for global EFT interpretations, the statistical principles and features of the tool are by no means limited to this type of application. In this talk, the EFTfitter.jl package is introduced and its statistical approach, features, and user interface are presented. Examples for the application of the tool in HEP are shown.

BAT.jl - Tutorial

• Oliver Schulz (Max Planck Society (DE))

BAT.jl is a Bayesian Analysis Toolkit in Julia. It is a high-performance tool box for Bayesian inference with statistical models expressed in a general-purpose programming language instead of a domain-specific language. Typical applications for this package are parameter inference given a model (in the form of a likelihood function and prior), the comparison of different models in the light of a given data set, and the test of the validity of a model to represent the data set at hand. BAT.jl provides access to the full Bayesian posterior distribution to enable parameter estimation, limit setting and uncertainty propagation. BAT.jl also provides supporting functionality like plotting recipes and reporting functions.

Analysis Grand Challenge with Julia

• Atell-Yehor Krasnopolski
• Jerry 🦑 Ling (Harvard University (US))

**Abstract.** The idea of the project [\[1\]][1] is to provide a new implementation of the IRIS-HEP Analysis Grand Challenge (AGC) [\[2\]][2] using the Julia programming language. The AGC is about performing the last steps in an analysis pipeline at scale to test workflows envisioned for the HL-LHC. This includes columnar data extraction from large datasets, processing of that data (event filtering, construction of observables, evaluation of systematic uncertainties) into histograms, statistical model construction and statistical inference, relevant visualisations for these steps, all done in a reproducible & preservable way that can scale to HL-LHC requirements. [\[3\]][3] [1]: https://github.com/Moelf/LHC_AGC.jl [2]: https://doi.org/10.5281/zenodo.8320276 [3]: https://agc.readthedocs.io/en/latest/

Using data frames in Julia to analyse HEP data

• Philippe Gras (Université Paris-Saclay (FR))

Data frames have been introduced for statistical analysis with the R language for dataset, much simpler and smaller than data encountered in HEP. A data frame based analysis consist in representing statistical variables in columns of a table, of which each row represents an observation. Julia offers a plethora of packages for data-frame based analysis that can work together thanks to the Julia multiple dispatch and the Table interface, defined in Tables.jl, they comply to. That includes UnROOT.jl, DataFrames.jl, TypedTables.jl, and DTables.jl. We propose in this talk to use the experience gained with the ROOT RDataFrame, the first HEP-specific data frame framework to draw guidelines for data frame based HEP data analysis with Julia and existing packages. We will also consider software development that would be needed to complement existing software.

Efficient computation of higher-order cumulants with Julia

• Piotr Kalaczyński (AstroCeNT)

Swift computation of cumulants is of importance in a number of fields, including HEP. There have been a number of implementations in other programming languages, however they have proven to be slow and limited only to low-order cumulants, while e.g., application in QCD measurements may require computation up to eighth-order. This challenge has been overcome by introducing a series of Julia libraries: Cumulants.jl, CumulantsFeature.jl, and CumulantsUpdates.jl, which allow calculation of multidimensional cumulants of an arbitrary order. Moreover, the libraries offer supreme computational speed, outrunning the analogous implementation in R by up to 2 orders in magnitude.

Using Symbolic Regression in Julia to find analytic funtions to model low statistics

• Guru Reddy (Kansas State University (US))

In High Energy Physics (HEP) analyses, low statistics often pose a challenge because a significant portion of events are discarded to define a region of interest. This can compromise the reliability of asymptotic limits in the CLs method. To address this, we employed Symbolic Regression in Julia to derive an analytical fit functions to accurately model our signal and background samples.

StochasticAD.jl: Derivatives of discrete randomness

• Gaurav Arya

Automatic differentiation (AD), a technique for constructing new programs which compute the derivative of an original program, has become ubiquitous in science due to the improved performance afforded by gradient-based optimization. However, a lot of scientific simulations, e.g. in particle physics, are discrete random. This motivates extending AD to handle discrete stochastic behaviors governed by distribution parameters, such as flipping a coin with probability p of being heads. These pose a challenge because the connection between the result (heads vs tails) and the parameters (p) is fundamentally discrete. This talk introduces StochasticAD.jl, a Julia AD system for handling discrete stochastic behaviours, and its application to discrete stochastic simulations in the sciences.

An Awkward module for round-tripping data structures between Python and Julia

• Ianna Osborne (Princeton University)

Both Julia and Python have a strong presence in the sciences. In a typical HEP data analysis process Python is more common, however, there is an obvious advantage to transitioning legacy software to Julia. We discuss the sharing of Awkward Array data structures between the two worlds to encourage the Python users to run their analysis both in an eco-system of their choice and in Julia. We discuss how the memory, the data buffer copies, and the dependencies are managed. We analyse the performance acceleration calling Julia from Python and vise versa for the intensive array-oriented calculations on a large scale, but not very large dimension arrays of HEP data.

HS3 - The HEP Statistics Serialization Standard

• Cornelius Grunwald (Technische Universitaet Dortmund (DE))
• Robin Pelkner (Technische Universitaet Dortmund (DE))

The HEP Statistics Serialization Standard (HS3) is an emerging standard providing a language-agnostic and software-independent format for saving statistical models of HEP in exchangeable files. HS3 allows publishing and sharing entire analyses in a JSON(-like) format that is both human and machine-readable. The standard aims to harmonize the use of statistical models across a range of different software frameworks, opening up new possibilities for validating results, reusing models in different contexts, and ensuring long-term readability. Efforts are currently underway to provide HS3 support for software commonly used in HEP, such as ROOT, Python, and Julia. In this talk, we give an introduction to the HS3 standard and discuss its design philosophy and semantics. We present the current status of the ongoing implementation of the HS3 interface in Julia and demonstrate first example usecases.

Advances in Amplitude Analysis with Julia

• Mikhail Mikhasenko (Excellence Cluster ORIGINS)

In the intricate realm of hadron physics, characterized by complex dynamics and a myriad of phenomena, there's a pressing need for tools that are both intuitive and robust. Enter the Julia language, celebrated for its adaptability and computational prowess. It has been adeptly utilized to craft libraries that delve deep into the nuances of particle decays, particularly those involving spin. The [`ThreeBodyDecay.jl`][1] package, tailored for three-body decays, showcases the elegance of Dalitz plots rendered in Julia. An intriguing aspect to note is the emergence of peaks and nodes in the decay probability distribution when particles with spin decay, a phenomenon that can be swiftly discerned with `ThreeBodyDecay.jl`. The recent synergy with SymPy in [`SymbolicThreeBodyDecays.jl`][2], facilitated by Julia's adept multiple dispatch mechanism, has revolutionized the capabilities of the module. Beyond the visual delight of inspecting decay models with seamlessly printed LaTeX expressions, there lies a significant advantage: the symbolic representation of the model. This approach not only foster model preservation but also paves the way for cross-language collaboration. It establishes a bridge between Julia and other programming languages and tools, such as `Python's JAX`, and anticipates integration with `RooFit`. This collaborative dimension accentuates Julia's potential to propel particle physics research, fostering cross-platform synergy. [1]: https://github.com/mmikhasenko/ThreeBodyDecay.jl [2]: https://github.com/mmikhasenko/SymbolicThreeBodyDecays.jl

Sensitivity of direct detection experiments to neutrino magnetic dipole moments

• Rocío Branada Balbontín (Universidad Técnica Federico Santa María)

With large active volume sizes dark matter direct detection experiments are sensitive to solar neutrino fluxes. Nuclear recoil signals are induced by 8B neutrinos, while electron recoils are mainly generated by the pp flux. Measurements of both processes offer an opportunity to test neutrino properties at low thresholds with fairly low backgrounds. In this work we study the sensitivity of these experiments to neutrino magnetic dipole moments assuming 1, 10 and 40 tonne active volumes (representative of XENON1T, XENONnT and DARWIN), 0.3 keV and 1 keV thresholds. We show that with nuclear recoil measurements alone a 40 tonne detector could be as competitive as Borexino, TEXONO and GEMMA, with sensitivities of order 8.0×10-11 µB at the 90% CL after one year of data taking. Electron recoil measurements will increase sensitivities way below these values allowing to test regions not excluded by astrophysical arguments. Using electron recoil data and depending on performance, the same detector will be able to explore values down to 4.0×10-12µB at the 90% CL in one year of data taking. By assuming a 200-tonne liquid xenon detector operating during 10 years, we conclude that sensitivities in this type of detectors will be of order 10-12 µB. Reducing statistical uncertainties may enable improving sensitivities below these values.

QED.jl - A Strong-field particle physics ecosystem

• Uwe Hernandez Acosta (Helmholtz-Zentrum Dresden-Rossendorf)

DAG Optimizations for Feynman Diagrams of High-Multiplicity Scattering Processes in Julia

• Anton Reinhard

The description of scattering processes in high energy physics is usually done with Feynman Diagrams. The number of Feynman Diagrams that can be generated for a given process explodes factorially with the number of particles. We discuss a possible approach enabling the calculation of higher-multiplicity scattering processes. We propose representing the calculation for a process as a directed acyclic graph (DAG) of small computation tasks. Using Julia, we can optimize this graph using subgraph replacement strategies together with an optimization algorithm. Finally, efficient code targeting arbitrary heterogeneous HPC systems can be generated from the optimized DAG.

Using Julia to Accelerate Monte Carlo Event Generation with Neural Importance Sampling

• Tom Jungnickel (CASUS / HZDR)

Monte Carlo event generation is essential for analysis in high energy physics and fast implementations are required to keep up with the large amounts of data measured by experiments. Therefore, these methods need to reflect the theoretical predictions accurately to enable efficient data generation, e.g. by rejection sampling. However, traditional importance sampling algorithms, such as the commonly used VEGAS algorithm, often struggle with adapting targets with multiple or non-coordinate aligned features, as is common in high energy physics. Especially in strong-field QED, processes dynamically depend on field parameters, which means the use of established codes for these problems needs to be questioned. An importance sampling approach using neural networks applied to strong-field processes is presented within the framework QED.jl. The quality of the generated proposals, e.g. the unweighting efficiency, is compared to VEGAS, providing insights beneficial to applications beyond strong-field QED.

Neurthino.jl - Propagating n-flavour neutrinos through Earth

• Johannes Schumann

In neutrino physics the determination of individual properties depends on neutrino oscillations. Neurthino.jl is a package for calculating neutrino oscillation probabilies not limited with respect to a specific n-flavour case. It also provides utilizes the information of the Preliminary Reference Earth Model (PREM) in order to easily setup baseline configurations with different electron densities along the Earths density profile.

Jet Finding in Julia

• Graeme A Stewart (CERN)

We describe the development and release of the new `JetReconstruction.jl` package, used for jet finding and clusterisation in HEP. This work grew out of a CHEP 2023 paper looking at language performance and ergonomics, where the Julia implementation impressed enough that it merited turning this into a production package. We outline the process of tuning and optimisation for these algorithms, within the Julia ecosystem, how different approaches were tried that resulted in the final implementations that are now available. Some important lessons that we learned, in terms of optimal algorithms and their realisation in Julia, are highlighted. Finally, we also describe the final interface design that is presented to users that allows internal development to continue, whilst providing a flexible and stable API.

Geant4.jl: Particle Transportation in Julia

• Pere Mato (CERN)

In this contribution I will be presenting the status of the Julia wrappers to the Geant4 simulation toolkit (https://github.com/JuliaHEP/Geant4.jl). The main emphasis will be the new ergonomics that Julia offers to particle transport toolkit, which is used basically by each HEP experiment, as well as in medical and space science. This enhanced toolkit usability is a great advantage compared with the C++ interface that sporadic users would need to use in order to perform their custom simulations.

A common interface for quadrivectors and particles

• Philippe Gras (Université Paris-Saclay (FR))

We propose a common interface for four-vectors and for elementary particle representation in the spirit of the Tables.jl package in order to ease interoperability between HEP packages.