PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
This addresses performance bottlenecks in multi-LLM serving systems for real-world applications, offering a novel optimization to reduce redundancy and improve efficiency.
The paper tackles the problem of redundant prefill stage execution and KV cache storage across multiple specialized language models in multi-agent systems, which increases latency and reduces throughput. The result is PrefillShare, a method that enables sharing the prefill module and KV cache, achieving 4.5x lower p95 latency and 3.9x higher throughput while matching full fine-tuning accuracy.
Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing across heterogeneous models in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads.