KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
This work addresses the practical challenge of efficient LLM serving for cloud providers, though it is incremental as it builds on existing KV caching methods with workload-specific optimizations.
The paper tackles the problem of optimizing KV cache performance for large language model serving by characterizing real-world workload patterns from a major cloud provider, finding that KV reuse patterns are skewed and predictable within request categories, and proposing a workload-aware cache eviction policy that improves serving performance under real-world traces.
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV\$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV\$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV\$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV\$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity.