KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
This addresses memory and bandwidth limitations for deploying efficient conversational AI systems, representing an incremental improvement with specific optimizations.
The paper tackles the efficiency bottleneck of increasing Key-Value (KV) cache during inference in large language models by proposing KV-Latent, which down-samples KV vector dimensions into a latent space, reducing KV cache footprint and improving inference speed with less than 1% extra training cost.
Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI. Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value (KV) cache during inference has emerged as a primary efficiency bottleneck, both in aspects of memory consumption and data transfer bandwidth limitations. To address these challenges, we propose a paradigm called KV-Latent. By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed, only with a small amount of extra training, less than 1\% of pre-training takes. Besides, we enhanced the stability of Rotary Positional Embedding applied on lower-dimensional vectors by modifying its frequency sampling mechanism, avoiding noise introduced by higher frequencies while retaining position attenuation. Our experiments, including both models with Grouped Query Attention and those without, have yielded satisfactory results. Finally, we conducted comparative experiments to study the impact of separately reducing Key and Value components on model's performance. Our approach allows for the construction of more efficient language model systems, and opens the new possibility on KV Cache saving and efficient LLMs. Our code is available at https://github.com/ShiLuohe/KV-Latent.