Recall Before You Rank: Similarity-Guided Top-$K$ Reuse for Efficient Long-Context Attention
This work addresses the efficiency bottleneck of sparse attention for long-context decoding, offering a training-free method that reduces selector cost while preserving accuracy, which is valuable for practitioners deploying large language models on long sequences.
ReTopK accelerates dynamic Top-K sparse attention by reusing historical retrieval decisions, achieving a 3.07x speedup over Exact Top-K at 128K context with only a 0.50% perplexity increase, while maintaining state-of-the-art performance on PG19, NIAH, and LongBench benchmarks.
Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top-$K$ selection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long-context decoding. In this paper, we introduce ReTopK, a training-free method that accelerates dynamic Top-$K$ attention by reusing historical retrieval decisions. ReTopK builds on the observation that similar queries often attend to overlapping supports and that partially overlapping supports can still preserve most of the Exact Top-$K$ attention mass. For each attention head, it maintains a bounded cache of historical query--support pairs, retrieves the most similar cached queries for each new query, unions their stored supports with a recent window, and reranks only the resulting compact candidate set using exact current-query scores. A similarity-based fallback invokes full-history Exact Top-$K$ when reuse is unreliable, while periodic exact refreshes limit cache drift. ReTopK retains the complete KV cache and reuses only selected indices, rather than historical scores, attention weights, or outputs. Across 16K--128K contexts, ReTopK achieves the lowest PG19 perplexity and the highest NIAH and LongBench scores among the evaluated approximate methods. At 128K with $K=512$, ReTopK incurs only a 0.50\% perplexity increase over Exact Top-$K$ while accelerating attention computation by $3.07\times$.