PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
For LLM serving systems, PEEK addresses the problem of inefficient KV cache utilization due to lack of awareness of future request prefixes, delivering substantial performance improvements in both online and offline settings.
PEEK introduces a scheduling and eviction framework that uses an incremental radix tree over the pending queue to expose prefix-sharing clusters, achieving up to 3.0× cache hit, 7.9× TTFT, 6.7× E2E, and 3.6× throughput gains over SGLang and vLLM baselines on workloads with exploitable prefix structure.
We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4$\times$H100 (DP=2 over TP=2), PEEK delivers up to 3.0$\times$/2.6$\times$ cache hit, 7.9$\times$/7.1$\times$ TTFT, 6.7$\times$/5.5$\times$ E2E, and 3.6$\times$/4.5$\times$ throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.