Epiphany-Aware KV Cache Eviction Without the Attention Matrix
For practitioners deploying long-context reasoning models, EpiKV provides a faster, attention-matrix-free cache eviction method that matches or exceeds prior attention-based approaches.
EpiKV introduces an epiphany score for KV cache eviction that avoids materializing the attention matrix, enabling use with fused kernels. At a 4096-token cache, it achieves 72% on MATH-500 (matching ThinKV at 71% and H2O at 67%), and at 8192 tokens on AIME-2024 reaches 37% (vs 33% baseline) with up to 2.8x speedup.
As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck. Existing cache eviction methods rank tokens by attention weight, which is a noisy importance proxy in long reasoning traces, and prohibits the use of fused kernels in production inference by forcing the model to materialize the attention matrix. In this work, we instead score tokens with a metric we term the epiphany score: the change in the model's internal representation, read directly from the forward pass with no attention matrix and negligible extra state. Our resulting cache eviction method, EpiKV, requires no training, classifier, or custom kernel, and can be used directly in FlashAttention inference stacks unchanged -- scaling to a 16x longer feasible context than attention-based scoring. upper-mid layers negatively) and remove a positional trend with a causal rolling z-score. At a 4096-token cache EpiKV reaches 72% on MATH-500, matching the strongest attention-based baseline (ThinKV 71%, H2O 67%); a lag-normalized KV variant reaches 37% on AIME-2024 at 8192 tokens against the best of them (33%), at up to 2.8x the speed.