What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs
This work provides a mechanistic understanding of a ubiquitous phenomenon in LLMs, which could lead to more efficient and effective pre-training for researchers and practitioners working with Transformer architectures.
The paper investigates why causal large language models consistently form an attention sink at position zero. They identify a 'P0-Sink Circuit' responsible for this behavior, showing it arises from causal attention's structural properties, not semantic content. The authors propose two parameter-free methods that accelerate P0 sink formation, outperforming the Transformer baseline and achieving performance comparable to Gated Attention.
Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position zero, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the P0-Sink Circuit, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at https://github.com/Pryest/flash-linear-attention.