LUMEN: Coordinated Failure Recovery for Distributed LLM Serving
For operators of large-scale LLM serving clusters, LUMEN addresses the critical problem of efficient failure recovery, which becomes increasingly important as cluster sizes grow.
LUMEN treats failure recovery in distributed LLM serving as a load-aware coordination problem, optimizing checkpoint placement, request redistribution, and capacity restoration to reduce serving and recovery times compared to existing approaches.
Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.