RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning
This addresses the 'impossible trinity' problem for researchers and practitioners using LLMs in domains like mathematics and programming, offering an incremental improvement over existing sparsity algorithms.
The paper tackles the inefficiency of LLMs in reasoning tasks by proposing RaaS, a sparsity-based algorithm that identifies milestone tokens to reduce time and memory complexities to O(L) while maintaining high accuracy, achieving results comparable to state-of-the-art methods with lower resource usage.
Large Language Models (LLMs) have demonstrated strong capabilities across various domains, with recent advancements in challenging reasoning tasks such as mathematics and programming. However, solving reasoning tasks often requires an LLM to generate long sequences, incurring $O(N)$ time and memory complexities per token, where $N$ is the current sequence length. To reduce complexities, existing sparsity-based algorithms propose to retain Key-Value (KV) vectors, the intermediate representations of only the most critical tokens. However, these algorithms struggle with the "impossible trinity" of accuracy, time, and memory. For example, the state-of-the-art algorithm, Quest, achieves high accuracy with $O(L)$ time but $O(N)$ memory ($L$ is the cache budget, $L \ll N$). To address the "impossible trinity", in this paper, we identify a new attention pattern during the decode stage of reasoning tasks, where milestone tokens (analogous to lemmas in mathematical proofs) emerge, are utilized, and then become unimportant afterward. Based on this pattern, we propose a new algorithm RaaS that identifies milestone tokens and retains their KV vectors until they are no longer needed, achieving high accuracy with $O(L)$ time and $O(L)$ memory complexities.