KVzap: Fast, Adaptive, and Faithful KV Cache Pruning
This addresses the inference speed and memory issues for users of large language models, though it appears incremental as an improvement over existing KV cache pruning methods.
The paper tackles the bottleneck of key-value (KV) cache in transformer-based language models by introducing KVzap, a fast and adaptive pruning method that achieves 2-4x KV cache compression with negligible accuracy loss on models like Qwen3-8B and Llama-3.1-8B-Instruct.
Growing context lengths in transformer-based language models have made the key-value (KV) cache a critical inference bottleneck. While many KV cache pruning methods have been proposed, they have not yet been adopted in major inference engines due to speed--accuracy trade-offs. We introduce KVzap, a fast, input-adaptive approximation of KVzip that works in both prefilling and decoding. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen3-32B across long-context and reasoning tasks, KVzap achieves $2$--$4\times$ KV cache compression with negligible accuracy loss and achieves state-of-the-art performance on the KVpress leaderboard. Code and models are available at https://github.com/NVIDIA/kvpress.