Rethinking the Role of Efficient Attention in Hybrid Architectures
For researchers designing efficient hybrid language models, this work provides mechanistic insights and a practical design principle to improve long-context performance.
The paper analyzes hybrid architectures combining full attention and efficient attention (e.g., sliding-window attention), finding that efficient attention primarily affects the speed of long-context capability emergence, while full attention carries long-range retrieval. It identifies 'Large-Window Laziness' and shows that removing positional encoding from full-attention layers in small-window hybrids improves long-context performance with minimal short-context impact.
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent sequence mixers. However, how these efficient modules shape model capabilities remains poorly understood. To address this gap, we conduct a systematic analysis across hybrid architectures from three perspectives: scaling behavior, mechanism analysis, and architecture design. First, from a scaling perspective, we find that efficient-attention design primarily affects how fast long-context capability emerges, while different hybrids eventually converge to comparable long-context performance under sufficient training. Second, mechanistically, we show that long-range retrieval is mainly carried by full attention, whereas efficient attention shapes its optimization trajectory. This explains a counter-intuitive phenomenon we call Large-Window Laziness: larger SWA windows can delay the formation of retrieval heads in full-attention layers. Third, guided by this mechanism, we show that applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance with negligible impact on short-context performance.