Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
For researchers interpreting Large Language Models, this provides a more faithful token importance metric and new mechanistic understanding of attention sinks.
The authors propose Contribution Weights, a projection-based metric that accounts for attention weight, value magnitude, and directional alignment, which outperforms attention-based metrics in identifying semantically critical tokens across decoder-only models. They also reveal that attention sinks actively suppress information and stabilize representations.
Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs). However, this approach has significant limitations as it neglects the geometric properties of the value vectors being aggregated. To address this gap, we introduce \emph{Contribution Weights}, a projection-based metric that quantifies a token's influence by accounting for it's attention weight, value magnitude, and directional alignment with the layer output. We demonstrate that contribution weights provide a more faithful measure of token importance, consistently outperforming attention-based metrics in identifying semantically critical tokens across different decoder-only models, tasks, and datasets. Further, our metric enables novel mechanistic analysis of \emph{attention sinks}. While previous work characterized sinks as passive repositories for excess attention, we reveal they serve an active functional role, suppressing information through a convex relationship between sink rate and output norm, stabilizing representations by opposing the semantic drift of low-confidence tokens.