Learning When Not to Attend Globally
This work addresses the computational bottleneck in LLM inference by enabling more efficient processing, though it is incremental as it builds on existing attention mechanisms.
The paper tackles the inefficiency of full attention in Large Language Models by proposing All-or-Here Attention (AHA), which dynamically switches between full and local sliding window attention, reducing up to 93% of attention operations with a 256-token window without performance loss.
When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs) can learn to dynamically determine when to attend to global context. We propose All-or-Here Attention (AHA), which utilizes a binary router per attention head to dynamically toggle between full attention and local sliding window attention for each token. Our results indicate that with a window size of 256 tokens, up to 93\% of the original full attention operations can be replaced by sliding window attention without performance loss. Furthermore, by evaluating AHA across various window sizes, we identify a long-tail distribution in context dependency, where the necessity for full attention decays rapidly as the local window expands. By decoupling local processing from global access, AHA reveals that full attention is largely redundant, and that efficient inference requires only on-demand access to the global context.