Adaptive KV-Cache Compression without Manually Setting Budget
This addresses memory footprint challenges in LLM inference for efficiency-critical applications, representing a novel method for a known bottleneck.
The paper tackles the memory inefficiency of KV-caches in large language model inference by introducing GVote, an adaptive compression scheme that eliminates manual budget setting, achieving up to 2× memory reduction while maintaining comparable or higher accuracy across benchmarks.
Large language models (LLMs) inference relies heavily on KV-caches to accelerate autoregressive decoding, but the resulting memory footprint grows rapidly with sequence length, posing significant efficiency challenges. Current KV-cache compression methods suffer from a Procrustes' bed problem: they force diverse workloads into fixed compression ratios, leading to suboptimal resource allocation and inference performance. To this end, we present GVote, an adaptive KV-cache compression scheme that eliminates manual budget specification while achieving superior accuracy-efficiency trade-offs. GVote operates on the principle that the important keys are the aggregation of keys required by future queries. The method predicts future query attention demands by Monte-Carlo style sampling potential queries and aggregating selected keys to determine the optimal cache budget without manual specification. Experimental evaluation demonstrates GVote's effectiveness across multiple benchmarks, including GSM8K, RULER and Longbench. Compared to baselines, GVote exhibits 2$\times$ memory reduction while the accuracy maintains higher or comparable.