EvolKV: Evolutionary KV Cache Compression for LLM Inference
This addresses memory bottlenecks in LLM inference for long-context tasks, offering a novel optimization approach that is incremental over existing heuristic methods.
The paper tackles the problem of inefficient key-value (KV) cache compression in large language model inference by proposing EvolKV, an adaptive framework that optimizes layer-wise cache allocation for memory efficiency and task performance, achieving up to 7 percentage point improvements on GSM8K and superior performance with only 1.5% of the original cache budget on code completion.
Existing key-value (KV) cache compression methods typically rely on heuristics, such as uniform cache allocation across layers or static eviction policies, however, they ignore the critical interplays among layer-specific feature patterns and task performance, which can lead to degraded generalization. In this paper, we propose EvolKV, an adaptive framework for layer-wise, task-driven KV cache compression that jointly optimizes the memory efficiency and task performance. By reformulating cache allocation as a multi-objective optimization problem, EvolKV leverages evolutionary search to dynamically configure layer budgets while directly maximizing downstream performance. Extensive experiments on 11 tasks demonstrate that our approach outperforms all baseline methods across a wide range of KV cache budgets on long-context tasks and surpasses heuristic baselines by up to 7 percentage points on GSM8K. Notably, EvolKV achieves superior performance over the full KV cache setting on code completion while utilizing only 1.5% of the original budget, suggesting the untapped potential in learned compression strategies for KV cache budget allocation.