NGT T1.2/1.3 <-> T2.2 - Transformer implementations

Friday 4 Jul 2025, 11:00 → 12:00 Europe/Zurich

61/1-007 - Room B (CERN)

11:00 → 11:10 Materials from existing works

Materials from hls4ml community on MHA and the adjacent FINN community.

In hls4ml, two MHA implementations have evolved over the years:
- An implementation from UW, as a prototype. Since developed and polished by Purdue group. Focuses on supporting "vanilla" MHA layer from Keras as a monolithic layer. Has scaling issues. Still not merged, mostly because of that.
- Implementation of quantized MHA from HGQ2, implemeted via einsum ops. Still not fully tested and reviewed by the community.

FINN team came up with their implementation of transformers with some constraints.

Current work has shifted towards efficient attention mechanisms. Purdue group pursues HEPT (training implementation here: https://github.com/Graph-COM/HEPT no public hls4ml implementation yet)
Fermilab group investigates SSMs as an alternative (mainly Mamba). Still unclear if they will pursue a FPGA implementation.

Speakers: Chang Sun (California Institute of Technology (US)), Dimitrios Danopoulos (CERN), Maria Carnesale (CERN), Michael Kagan (SLAC National Accelerator Laboratory (US)), Nadezhda Nikolaeva Dobreva (Nikhef National institute for subatomic physics (NL)), Nivedaa Dhandapani 🐥 (University of Massachusetts (US)), Rimsky Alejandro Rojas Caballero (CERN), Roope Oskari Niemi, Sebastian Dittmeier (Ruprecht-Karls-Universitaet Heidelberg (DE)), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Vladimir Loncar (CERN)

FINN-T Compiling Custom Dataflow Accelerators for Quantized Transformers.pdf

hgq-hls4ml-xformer.pdf

HLS4ML-Transformer-presentation.pdf

LayerNorm and MHA in hls4ml.pdf

11:10 → 12:00 Discussion

Speakers: Chang Sun (California Institute of Technology (US)), Dimitrios Danopoulos (CERN), Maria Carnesale (CERN), Michael Kagan (SLAC National Accelerator Laboratory (US)), Nadezhda Nikolaeva Dobreva (Nikhef National institute for subatomic physics (NL)), Nivedaa Dhandapani 🐥 (University of Massachusetts (US)), Rimsky Alejandro Rojas Caballero (CERN), Roope Oskari Niemi, Sebastian Dittmeier (Ruprecht-Karls-Universitaet Heidelberg (DE)), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Vladimir Loncar (CERN)

GMT20250704-090711_Recording_1920x1080.mp4

Introduction to (previous) work.pdf