Interpreting Transformers for Jet Tagging
This work provides incremental insights into transformer models for domain-specific jet-tagging tasks in high-energy physics.
The study tackled the problem of interpreting the Particle Transformer (ParT) model for jet tagging in particle physics by analyzing attention patterns and correlations, revealing a binary attention pattern and that the model learns traditional jet substructure observables.
Machine learning (ML) algorithms, particularly attention-based transformer models, have become indispensable for analyzing the vast data generated by particle physics experiments like ATLAS and CMS at the CERN LHC. Particle Transformer (ParT), a state-of-the-art model, leverages particle-level attention to improve jet-tagging tasks, which are critical for identifying particles resulting from proton collisions. This study focuses on interpreting ParT by analyzing attention heat maps and particle-pair correlations on the $η$-$φ$ plane, revealing a binary attention pattern where each particle attends to at most one other particle. At the same time, we observe that ParT shows varying focus on important particles and subjets depending on decay, indicating that the model learns traditional jet substructure observables. These insights enhance our understanding of the model's internal workings and learning process, offering potential avenues for improving the efficiency of transformer architectures in future high-energy physics applications.