The computation of higher-order derivatives using algorithmic differentiation has been an active area of research for several decades. In recent years, this topic has gained renewed attention due to the growing success of physics-informed neural networks (PINNs), in which such derivatives must be efficiently propagated through corresponding networks.
In this talk, we discuss several...
Computing partial differential equation (PDE) operators via nested backpropagation is expensive, yet popular, and severely restricts their utility for scientific machine learning. Recent advances, like the forward Laplacian and randomizing Taylor mode automatic differentiation (AD), propose forward schemes to address this. We introduce an optimization technique for Taylor mode that "collapses"...
Resolving a long-standing open question, we show that that the core AD problem of accumulating a Jacobian matrix while minimizing multiplications is NP-complete. Complementing this, we show that the running time of a relatively straight-forward $O^*(2^n)$ algorithm is essentially best possible under the Exponential Time Hypothesis. We also establish NP-completeness for the 'scarcity' problem...
Computer simulations of the solution to conservation laws are important for the analysis of fluid flows, which is in turn used in the design of aircraft and spacecraft. Perhaps the most famous of these conservation laws are the Navier-Stokes equations, which describe the flow of most real fluids. If one considers an inviscid fluid, the Navier-Stokes equations reduce to the Euler equations,...
The talk discusses the Abs-Normal Form as a structured representation of piecewise linear functions and examines its main advantages and limitations. This form provides a systematic way to capture piecewise linear behavior commonly encountered in Algorithmic Piecewise Differentiation. Alternative representations, including the ReLU Normal Form and the more general Piecewise Normal Form, are...
While there are a few common variants to restrict a value to a range, such as a clamp function and a smoothstep, they commonly have regions with vanishing gradients outside the target range. This makes these methods non-optimal for differentiable programming. Thus, we have built a better “clamp” using periodic functions, chosen to be efficient to compute compared to similar alternatives....
In this talk I will present my ongoing research in differentiable programming, specifically its application to scenarios with discontinuities induced by discrete structural constructs like conditional statements. If the parameters with respect to which we wish to differentiate appear in the predicate, then obtaining derivatives with respect to those parameters by propagating tangents or...
We introduce qruise-toolset, a differentiable quantum simulation toolbox with a Python interface and a Julia simulation backend. It is specifically tailored for simulating open quantum systems and solving quantum optimal control problems. The toolset allows the user to benefit from the performance that the Julia JIT compilation offers and still feel at home with the convenience that the Python...
Stochastic rounding is a pivotal technique to perform scientific computation in reduced precision and is becoming available on modern hardware. This talk will briefly introduce stochastic rounding and demonstrate automatic differentiation on several case studies using a software implementation of stochastic rounding in Julia.
Clad is a Clang plugin that enables automatic differentiation for C++ by transforming abstract syntax trees using LLVM's compiler infrastructure. A key design goal is generating readable, efficient derivatives that integrate seamlessly into existing codebases. This talk explores the challenges of extending Clad to support object-oriented programming, demonstrating our approach through examples...
pythonOCC is an open-source library extension that provides Python bindings for the widely-used Open CASCADE Technology (OCCT) geometry modeling kernel. It significantly facilitates the use of the CAD kernel in the context of automated processes for multidisciplinary design analysis and optimization (MDAO). To support gradient-based shape optimization, pythonOCC was differentiated using the AD...
In operator overloading algorithmic differentiation tools, data is recorded on the tape for the reverse mode. The order of the statements is defined by the program, and the AD tool has to make an educated guess on the best layout of the adjoints, e.g., identifiers for the values. Since they are called for every floating-point operation, the heuristics need to be quick. In this talk, we want to...
We describe ADTAYL, a Matlab package in two parts. The first, adtayl, is a class whose objects are arrays that behave like native Matlab arrays, except that each array element is a one-variable Taylor series (TS) instead of a number.
A TS is really a polynomial, but we call it TS because arithmetic operations don't quite follow the rules for algebra of polynomials. There is a maximum...
We present a differentiation framework for plane-wave density-functional theory (DFT) that combines the strengths of forward-mode algorithmic differentiation (AD) and density-functional perturbation theory (DFPT). In the resulting AD-DFPT framework derivatives of any DFT output quantity with respect to any input parameter (e.g. geometry, density functional or pseudopotential) can be computed...
Simulating two-phase flow in porous media requires solving large nonlinear systems, commonly via Newton–Krylov methods such as GMRES. These methods rely on Jacobian–vector products, which can be efficiently computed using automatic differentiation (AD), avoiding explicit Jacobian assembly. However, the lack of an assembled Jacobian complicates preconditioner design.
This work presents a...
Applying automatic differentiation (AD) to particle simulations such as Geant4 opens the possibility of addressing optimization tasks in high energy physics, such as guiding detector design and parameter fitting, with powerful gradient-based optimization methods. In this talk, we refine our previous work on differentiable simulation with Geant by incorporating multiple coulomb scattering into...
The Generalized Truncated Power Series Algebra (GTPSA) is a high-order differential algebra framework that provides real and complex multivariate Taylor expansions with accuracy close to symbolic computation. It naturally supports automatic differentiation of abritrary order, with simultaneous propagation of mixed partial derivatives and functional composition, while maintaining high...
Teaching is not only an integral part of everyday academic life, but also a key opportunity for academic teaching staff to recruit young scientists.
To that end, illustrative assignments that allow students to actively engage with topics relevant to current research are especially valuable.
We present a classroom-tested assignment for undergraduate students in their sophomore year. In...
I have been teaching AD etc. (numerical methods, programming, compilers) to students at various levels of academic maturity (ranging from 1st year BSc to PhD) for a few years now. Many topics covered feature considerable overlap in terms of data structures (e.g., matrices, graphs) and algorithms (e.g., basic linear algebra, graph traversal / transformation) appearing at the very core of the...
The compiler research group pioneered interactive C++ notebooks with xeus-clang-repl, and its successor xeus-cpp. The ability to write automatic differentiation code in an interactive context eliminates the need for long edit-compile-run cycles and simplifies the approach to teaching computational methods.
By leveraging the clang-repl C/C++ interpreter, we create an interactive notebook...
Elimination techniques for computational graphs can substantially speed up the accumulation of Jacobian matrices. Low-level elimination techniques that operate on the level of elementary operations are already integrated in various AD tools and can automatically speed up derivative calculations for the user. High-level elimination techniques, like generalized face elimination, currently...
Bidirectional random number generators (RNGs) allow stochastic sequences to be reproduced not only forward but also backward in time. This capability can be leveraged in adjoint automatic differentiation (AD) to significantly reduce memory usage: instead of storing all intermediate random variates on the tape for backpropagation, the AD engine can efficiently regenerate them in reverse order...
Designing optics for accelerators is a continuous process requiring the solution of many multi-dimensional optimization problems. Given the multitude of operating configurations that have to be considered, as well as the increasing size and complexity of modern accelerators, runtime becomes a limiting factor — in particular because computing the required derivatives accounts for most of the...
The HFBTHO code implements a nuclear energy density functional solver to model the structure of atomic nuclei. HFBTHO has previously been used to calibrate energy functionals and perform sensitivity analysis by using derivative-free methods. To enable derivative-based optimization and uncertainty quantification approaches, we must compute the derivatives of HFBTHO outputs with respect to the...
Every summer, the [Institute of Computing for Climate Science (ICCS)][1] hosts a summer school on software engineering and scientific computing for climate science. The target audience mainly includes students and scientists that we collaborate with on climate modelling projects and the aim is to provide them training on software engineering best practices and updates on current topics....
