Speaker
Description
Transformers have gained significant traction in high-energy physics (HEP) experiments. However, many HEP applications require ultra-fast processing and the quadratic complexity of standard self-attention becomes prohibitive. Fortunately, there are several approaches to address this bottleneck. One is to replace this standard self-attention with linearized attention. In this work, we explore one such mechanism, Linformer, which approximates self-attention through low-rank projections, reducing the attention complexity from quadratic to linear in the number of input elements. We train a transformer model with Linformer for particle reconstruction using a public Compact Linear Collider dataset. We compare its performance against the baseline reconstruction algorithm, Pandora, and demonstrate that this approach maintains competitive reconstruction performance. To isolate the effect of linearization, we also compare this model against a model with the same architecture but uses a quadratic complexity attention, and characterize the trade-off between attention complexity and reconstruction performance.