15–18 Oct 2024
Purdue University
America/Indiana/Indianapolis timezone

Interpreting and Accelerating Transformers for Jet Tagging

15 Oct 2024, 15:25
5m
Steward Center 306 (Third floor) (Purdue University)

Steward Center 306 (Third floor)

Purdue University

128 Memorial Mall Dr, West Lafayette, IN 47907
Lightning 5 min talk + poster Lighting talks

Speakers

Aaron Wang (University of Illinois at Chicago (US)) Vivekanand Gyanchand Sahu (University of California San Diego)

Description

Attention-based transformers are ubiquitous in machine learning applications from natural language processing to computer vision. In high energy physics, one central application is to classify collimated particle showers in colliders based on the particle of origin, known as jet tagging. In this work, we study the interpretatbility and prospects for acceleration of Particle Transformer (ParT), a state-of-the-art model, leverages particle-level attention to improve jet-tagging performance. We analyzing ParT's attention maps and particle-pair correlations in the $\eta$-$\phi$ plane, revealing intriguing features, such as a binary attention pattern that identifies critical substructure in jets. These insights enhance our understanding of the model's internal workings and learning process and hint at ways to improve its efficiency. Along these lines, we also explore low-rank attention, attention alternatives, and dynamic quantization to accelerate transformers for jet tagging. With quantization, we achieve a 50% reduction in model size and a 10% increase in inference speed without compromising accuracy. These combined efforts enhance both the performance and the interpretability of transformers in high-energy physics, opening avenues for more efficient and physics-driven model designs.

Focus areas HEP

Authors

Aaron Wang (University of Illinois at Chicago (US)) Abhijith Gandrakota (Fermi National Accelerator Lab.(US)) Elham Khoda (University of Washington (US)) Javier Mauricio Duarte (Univ. of California San Diego (US)) Jennifer Ngadiuba (FNAL) Vivekanand Gyanchand Sahu (University of California San Diego)

Presentation materials