From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
For researchers and engineers designing LLM serving systems, this survey provides a structured taxonomy and identifies gaps in current evaluation practices.
This survey classifies over 30 KV cache management systems for LLM serving using four axes (locality, lifetime, ownership, substrate), revealing five architectural archetypes. It identifies that ownership is a key differentiator in distributed systems and highlights seven missing KV-specific measurements linked to open problems.
The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes -- local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current evaluations and identifies seven missing KV-specific measurements, linking them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.