KVpop -- Key-Value Cache Compression with Predictive Online Pruning
For LLM practitioners, KVpop reduces memory costs in autoregressive decoding while maintaining high output quality, addressing a key bottleneck in long-context inference.
KVpop introduces a learned KV cache eviction policy that uses a future-attention target for supervision, achieving 98% of full-attention performance at 75% compression and 97% at 88% compression on mathematical reasoning tasks, outperforming existing baselines.
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.