Towards Resiliency in Large Language Model Serving with KevlarFlow
This addresses service availability issues for users of hyperscale LLM serving systems, representing a strong incremental improvement over existing recovery mechanisms.
The paper tackles the problem of frequent hardware faults causing service outages in large language model serving systems by introducing KevlarFlow, a fault-tolerant architecture that reduces mean-time-to-recovery by 20x and improves latency and time-to-first-token metrics by up to 574.6x under failure conditions.
Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow, a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems.