Recency/Frequency Adaptive KV Caching for Large Language Model Serving
For LLM serving systems, this work improves cache efficiency and latency, though the gains on real-world workloads are incremental.
The paper tackles the problem of inefficient KV caching in LLM serving due to unrelated workloads flushing each other's caches. It proposes an adaptive caching method that dynamically allocates cache space between recent and frequent KV blocks, achieving up to 10.8% higher cache hit rate and 12.6% lower time-to-first-token on synthetic workloads, and 2.1% and 2.0% improvements on real-world workloads.
Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% over naive vLLM on synthetic document question answering workloads, and 2.1% and 2.0% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.