From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
This work addresses performance enhancement for long-context language models, but it is incremental as it builds on existing mechanistic insights.
The paper tackled the problem of leveraging retrieval heads to improve long-context capabilities in LLMs, achieving a +2.28 point gain on HELMET at 128K for Llama-3.1, with +70% improvement in citation generation and +32% in passage re-ranking.
Advances in mechanistic interpretability have identified special attention heads, known as retrieval heads, that are responsible for retrieving information from the context. However, the role of these retrieval heads in improving model performance remains unexplored. This work investigates whether retrieval heads can be leveraged to enhance the long-context capabilities of LLMs. Specifically, we propose RetMask, a method that generates training signals by contrasting normal model outputs with those from an ablated variant in which the retrieval heads are masked. This mechanism-based approach achieves substantial improvements: +2.28 points on HELMET at 128K for Llama-3.1, with +70% gains on generation with citation and +32% on passage re-ranking, while preserving performance on general tasks. Experiments across three model families reveal that the effectiveness depends on retrieval head organization: models with concentrated patterns of retrieval heads respond strongly, while those with distributed patterns show limited gains. This mechanistic relationship validates the function of retrieval heads and demonstrates that mechanistic insights can be transformed into performance enhancements.