MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
For practitioners deploying multimodal LLMs with limited memory, this method improves KV cache efficiency without retraining, but the improvement is incremental over existing prefill-stage selection techniques.
MM-ShiftKV addresses the problem that prefill-stage KV selection methods fail in multimodal LLMs because decoding-time queries have higher variance than prefilling representations, causing unstable importance estimation. The proposed training-free method constructs variance-expanded query proxies during prefilling to better estimate KV importance, consistently outperforming existing methods under strict cache budgets.
Key-Value (KV) caching is essential for efficient inference in multimodal large language models (MLLMs), yet its memory footprint grows linearly with context length and becomes a major bottleneck due to the large number of visual tokens. Recent prefill-stage KV selection methods estimate KV importance from prefilling statistics, implicitly assuming that prefilling-time queries are representative of those encountered during decoding. We show that this assumption breaks down in multimodal inference, where decoding-time queries exhibit substantially larger variance than prefilling-stage representations, leading to unstable KV importance estimation under tight cache budgets. As a result, small ranking errors can disproportionately discard semantically critical visual tokens and degrade grounding and reasoning performance. We propose MM-ShiftKV, a training-free, decode-aware and strictly prefill-only KV selection method. MM-ShiftKV approximates decoding-time query behavior during prefilling by constructing variance-expanded query proxies and estimates prompt KV importance based on their aggregated attention mass. Experiments on multimodal benchmarks demonstrate that MM-ShiftKV consistently outperforms existing methods under strict KV-cache budgets. Our code is available at https://github.com/zjuDBxAI/MM-ShiftKV.