Name: NGT T1.2/1.3 T2.2 - Transformer implementations
Start: 2025-07-04T11:00:00+02:00
End: 2025-07-04T12:00:00+02:00
Location: CERN

- 1
  
  Materials from existing works
  
  Materials from hls4ml community on MHA and the adjacent FINN community.
  
  In hls4ml, two MHA implementations have evolved over the years:
  - An implementation from UW, as a prototype. Since developed and polished by Purdue group. Focuses on supporting "vanilla" MHA layer from Keras as a monolithic layer. Has scaling issues. Still not merged, mostly because of that.
  - Implementation of quantized MHA from HGQ2, implemeted via einsum ops. Still not fully tested and reviewed by the community.
  
  FINN team came up with their implementation of transformers with some constraints.
  
  Current work has shifted towards efficient attention mechanisms. Purdue group pursues HEPT (training implementation here: https://github.com/Graph-COM/HEPT no public hls4ml implementation yet)
  Fermilab group investigates SSMs as an alternative (mainly Mamba). Still unclear if they will pursue a FPGA implementation.
  
  Speakers: Chang Sun (California Institute of Technology (US)), Dimitrios Danopoulos (CERN), Maria Carnesale (CERN), Michael Kagan (SLAC National Accelerator Laboratory (US)), Nadezhda Nikolaeva Dobreva (Nikhef National institute for subatomic physics (NL)), Nivedaa Dhandapani 🐥 (University of Massachusetts (US)), Rimsky Alejandro Rojas Caballero (CERN), Roope Oskari Niemi, Sebastian Dittmeier (Ruprecht-Karls-Universitaet Heidelberg (DE)), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Vladimir Loncar (CERN)
  
  FINN-T Compiling Custom Dataflow Accelerators for Quantized Transformers.pdf
  
  hgq-hls4ml-xformer.pdf
  
  HLS4ML-Transformer-presentation.pdf
  
  LayerNorm and MHA in hls4ml.pdf
- 2
  
  Discussion
  
  Speakers: Chang Sun (California Institute of Technology (US)), Dimitrios Danopoulos (CERN), Maria Carnesale (CERN), Michael Kagan (SLAC National Accelerator Laboratory (US)), Nadezhda Nikolaeva Dobreva (Nikhef National institute for subatomic physics (NL)), Nivedaa Dhandapani 🐥 (University of Massachusetts (US)), Rimsky Alejandro Rojas Caballero (CERN), Roope Oskari Niemi, Sebastian Dittmeier (Ruprecht-Karls-Universitaet Heidelberg (DE)), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Vladimir Loncar (CERN)
  
  GMT20250704-090711_Recording_1920x1080.mp4
  
  Introduction to (previous) work.pdf

NGT T1.2/1.3 <-> T2.2 - Transformer implementations

61/1-007 - Room B

CERN