KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
This addresses the memory and latency bottleneck in LLM deployment for efficient inference, representing a novel method for a known bottleneck.
The paper tackles the problem of KV cache memory bottleneck in large language model inference by proposing Coupled Quantization, which exploits inter-dependencies between channels to compress the cache down to 1-bit per channel while preserving model quality, outperforming or matching existing baselines.
Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As the batch size, context length, or model size increases, the size of the key and value (KV) cache can quickly become the main contributor to GPU memory usage and the bottleneck of inference latency. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail at very low bit widths. We observe that distinct channels of a key/value activation embedding are highly inter-dependent, and the joint entropy of multiple channels grows at a slower rate than the sum of their marginal entropies. Based on this insight, we propose Coupled Quantization (CQ), which couples multiple key/value channels together to exploit their inter-dependency and encode the activations in a more information-efficient manner. Extensive experiments reveal that CQ outperforms or is competitive with existing baselines in preserving model quality. Furthermore, we demonstrate that CQ can preserve model quality with KV cache quantized down to 1-bit.