Materials from hls4ml community on MHA and the adjacent FINN community.
In hls4ml, two MHA implementations have evolved over the years:
- An implementation from UW, as a prototype. Since developed and polished by Purdue group. Focuses on supporting "vanilla" MHA layer from Keras as a monolithic layer. Has scaling issues. Still not merged, mostly because of that.
- Implementation of quantized MHA from HGQ2, implemeted via einsum ops. Still not fully tested and reviewed by the community.
FINN team came up with their implementation of transformers with some constraints.
Current work has shifted towards efficient attention mechanisms. Purdue group pursues HEPT (training implementation here: https://github.com/Graph-COM/HEPT no public hls4ml implementation yet)
Fermilab group investigates SSMs as an alternative (mainly Mamba). Still unclear if they will pursue a FPGA implementation.
Speakers:
Chang Sun
(California Institute of Technology (US)),
Dimitrios Danopoulos
(CERN),
Maria Carnesale
(CERN),
Michael Kagan
(SLAC National Accelerator Laboratory (US)),
Nadezhda Nikolaeva Dobreva
(Nikhef National institute for subatomic physics (NL)),
Nivedaa Dhandapani 🐥
(University of Massachusetts (US)),
Rimsky Alejandro Rojas Caballero
(CERN),
Roope Oskari Niemi,
Sebastian Dittmeier
(Ruprecht-Karls-Universitaet Heidelberg (DE)),
Verena Ingrid Martinez Outschoorn
(University of Massachusetts (US)),
Vladimir Loncar
(CERN)