Identifying and Evaluating Inactive Heads in Pretrained LLMs
This work addresses efficiency issues in LLMs for practitioners by revealing significant computational waste, though it is incremental as it builds on known attention sink phenomena.
The researchers tackled the problem of computational redundancy in pretrained LLMs by identifying inactive attention heads, finding that more than 12% of heads are inactive on average and can be ablated while maintaining MMLU accuracy within 1% of the original model. They developed a taxonomy of 13 score functions to measure inactivity, showing that methods relying solely on attention weights underestimate inactive heads by over 7%.
Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behaviors like attention sinks, where the first token receives the most attention despite limited semantic importance, suggest some heads may be inactive, and point to a significant source of computational redundancy. To analyze this phenomenon, we propose a taxonomy of 13 score functions that measure different ways a head can be inactive. Thresholding these scores allows us to analyze different sets of potentially inactive attention heads. We evaluate whether identified heads are inactive through model interventions, finding that more than 12% of attention heads are inactive on average, and can be ablated in specific contexts while maintaining MMLU accuracy to within 1% of the pretrained LLM. Across 3 model families, our score functions that measure the average norm of a head's output consistently identify inactive heads that would not have been found by score functions that rely solely on attention weights. We establish that relying on a score function that measures a first token attention sink would underestimate the prevalence of inactive heads, failing to identify more than 7% of inactive heads on average. We also show how measuring score distributions can provide insights into attention behavior. For instance, we find evidence that finetuning causes little to no change in attention behavior, and that even within the same model family, large model scales present markedly different attention behaviors.