PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
This addresses memory bottlenecks for real-time LLM applications like chatbots, representing an incremental improvement over existing KV cache compression techniques.
The paper tackles the problem of high GPU memory usage during LLM inference by proposing PyramidInfer, a method that compresses the KV cache through layer-wise retention of crucial context, resulting in a 2.2x throughput improvement and over 54% GPU memory reduction compared to baseline methods.
Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.