Support Basis: Fast Attention Beyond Bounded Entries
This addresses a central scaling problem for large language models, offering a practical improvement over prior methods that were limited by unrealistic assumptions.
The paper tackles the quadratic complexity bottleneck of softmax attention in large language models by introducing a support-basis decomposition framework that works beyond restrictive bounded-entry assumptions, achieving sub-quadratic runtime with theoretical guarantees and empirical validation on sub-Gaussian data.
The quadratic complexity of softmax attention remains a central bottleneck in scaling large language models (LLMs). [Alman and Song, NeurIPS 2023] proposed a sub-quadratic attention approximation algorithm, but it works only under the restrictive bounded-entry assumption. Since this assumption rarely holds in practice, its applicability to modern LLMs is limited. In this paper, we introduce support-basis decomposition, a new framework for efficient attention approximation beyond bounded entries. We empirically demonstrate that the entries of the query and key matrices exhibit sub-Gaussian behavior. Our approach uses this property to split large and small entries, enabling exact computation on sparse components and polynomial approximation on dense components. We establish rigorous theoretical guarantees, proving a sub-quadratic runtime, and extend the method to a multi-threshold setting that eliminates all distributional assumptions. Furthermore, we provide the first theoretical justification for the empirical success of polynomial attention [Kacham, Mirrokni, and Zhong, ICML 2024], showing that softmax attention can be closely approximated by a combination of multiple polynomial attentions with sketching.