SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
This addresses the issue of information loss and performance degradation in large language model inference for users needing efficient deployment, though it is incremental as it builds on existing compression methods.
The paper tackled the problem of semantic fragmentation in KV cache compression by introducing SemantiCache, which preserves semantic integrity through semantic chunking and clustered merging, resulting in up to 2.61 times faster decoding and reduced memory footprint while maintaining comparable performance.
Existing KV cache compression methods generally operate on discrete tokens or non-semantic chunks. However, such approaches often lead to semantic fragmentation, where linguistically coherent units are disrupted, causing irreversible information loss and degradation in model performance. To address this, we introduce SemantiCache, a novel compression framework that preserves semantic integrity by aligning the compression process with the semantic hierarchical nature of language. Specifically, we first partition the cache into semantically coherent chunks by delimiters, which are natural semantic boundaries. Within each chunk, we introduce a computationally efficient Greedy Seed-Based Clustering (GSC) algorithm to group tokens into semantic clusters. These clusters are further merged into semantic cores, enhanced by a Proportional Attention mechanism that rebalances the reduced attention contributions of the merged tokens. Extensive experiments across diverse benchmarks and models demonstrate that SemantiCache accelerates the decoding stage of inference by up to 2.61 times and substantially reduces memory footprint, while maintaining performance comparable to the original model.