Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference
This addresses performance bottlenecks in LLM inference for applications like RAG and agents, but is incremental as it builds on existing caching concepts.
The paper tackles the problem of inefficient Key-Value Cache (KVC) management for large language model inference with extended context windows, analyzing real-world access patterns and evaluating commercial systems to demonstrate the lack of tailored solutions and provide design insights for scalable, low-latency systems.
The increasing adoption of large language models (LLMs) with extended context windows necessitates efficient Key-Value Cache (KVC) management to optimize inference performance. Inference workloads like Retrieval-Augmented Generation (RAG) and agents exhibit high cache reusability, making efficient caching critical to reducing redundancy and improving speed. We analyze real-world KVC access patterns using publicly available traces and evaluate commercial key-value stores like Redis and state-of-the-art RDMA-based systems (CHIME [1] and Sherman [2]) for KVC metadata management. Our work demonstrates the lack of tailored storage solution for KVC prefilling, underscores the need for an efficient distributed caching system with optimized metadata management for LLM workloads, and provides insights into designing improved KVC management systems for scalable, low-latency inference.