KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
This addresses a major bottleneck in efficient long-context LLM deployment, offering a practical and scalable solution for inference pipelines.
The paper tackles the problem of KV cache size growth in large language models during long-context inference by proposing a compression framework using attention-guided, layer-adaptive composite tokens, achieving significant memory reduction while preserving accuracy and outperforming prior methods.
Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding; however, cache size grows linearly with context length and model depth, becoming a major bottleneck in long-context inference. Prior KV cache compression methods either enforce rigid heuristics, disrupt tensor layouts with per-attention-head variability, or require specialized compute kernels. We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens. Our method aggregates attention scores to estimate token importance, selects head-specific tokens independently, and aligns them into composite tokens that respect the uniform cache structure required by existing inference engines. A global allocation mechanism further adapts retention budgets across layers, assigning more capacity to layers with informative tokens. This approach achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods. Crucially, our approach remains fully compatible with standard inference pipelines, offering a practical and scalable solution for efficient long-context LLM deployment.