SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
This addresses efficiency issues for users deploying LLMs in tasks like multi-document summarization and conversational agents, though it is an incremental improvement over existing cache compression methods.
The paper tackles the memory bottleneck of key-value caches in large language model inference by proposing SemShareKV, a framework that shares caches across semantically similar prompts using token-level LSH matching, achieving up to 6.25x speedup and 42% lower GPU memory usage with minimal quality loss.
As large language models (LLMs) continue to scale, the memory footprint of key-value (KV) caches during inference has become a significant bottleneck. Existing approaches primarily focus on compressing KV caches within a single prompt or reusing shared prefixes or frequently ocurred text segments across prompts. However, such strategies are limited in scenarios where prompts are semantically similar but lexically different, which frequently occurs in tasks such as multi-document summarization and conversational agents. We propose \textit{SemShareKV}, a KV cache sharing and compression framework that accelerates LLM inference by reusing KVCache in semantically similar prompts. Instead of relying on exact token matches, SemShareKV applies fuzzy token matching using locality-sensitive hashing (LSH) on token embeddings and incorporates Rotary Position Embedding (RoPE) to better preserve positional information. By selectively reusing relevant key-value pairs from a reference prompt's cache, SemShareKV reduces redundant computation while maintaining output quality. Experiments on diverse summarization datasets show up to 6.25$\times$ speedup and 42\% lower GPU memory usage with 5k tokens input, with negligible quality degradation. These results highlight the potential of semantic-aware cache sharing for efficient LLM inference.