Still: Amortized KV Cache Compaction in a Single Forward Pass
For practitioners deploying long-context language models, Still provides a practical, reusable cache compaction method that is both fast and expressive, addressing a key memory bottleneck.
Still introduces a lightweight, train-once Perceiver module that compresses KV caches in a single forward pass, achieving state-of-the-art speed-quality trade-offs across 8x to 200x compression ratios and 8k to 128k context lengths, outperforming baselines by 8-22 points on RULER and enabling iterative compression for long-horizon deployment.
The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive enough to preserve context under constraint, and reusable across a trajectory. Existing compaction methods satisfy only part of this requirement: selection methods are lightweight but subset-bound, while synthesis methods are expressive but rely on per-context optimization. Here we introduce Still, a small per-layer Perceiver trained once against a frozen base model that produces compact keys and values in a single forward pass. On Qwen and Gemma models, Still occupies the favorable side of the speed--quality frontier across compression ratios from $8\times$ to $200\times$ and context lengths from $8$k to $128$k. On the long-context RULER grid, Still exceeds the strongest baseline by 8--22 points. The same compact cache also supports free-form summarization, preserving most of the full-context gain on HELMET and winning a pairwise LongBench summarization comparison against KV-Distill. Because compaction is a forward pass, Still can be applied iteratively, entering a long-horizon regime unavailable to per-context methods. We show that amortization makes long-context cache compaction tractable, and synthesis makes its compact state useful at extreme compression.