Block: Balancing Load in LLM Serving with Context, Knowledge and Predictive Scheduling
This addresses the problem of inefficient scheduling in LLM serving systems for users and providers, offering a scalable solution with significant performance improvements, though it is incremental as it builds on existing scheduling concepts.
The paper tackles load balancing and auto-provisioning in large language model serving by introducing Block, a distributed scheduling framework that uses contextual information from requests to predict metrics, resulting in up to 16.7% higher serving capacity and up to 49.5% lower P99 tail latency compared to heuristic schedulers.
This paper presents Block, a distributed scheduling framework designed to optimize load balancing and auto-provisioning across instances in large language model serving frameworks by leveraging contextual information from incoming requests. Unlike popular model serving systems that rely on monolithic and heuristic task schedulers, Block operates as a fully distributed, stateless, and predictive scheduling system to achieve low overhead, reliability, and scalability. It leverages the deterministic and predictable characteristics of LLM inferences, such as host configurations, response lengths, and hardware performance, to make scheduling decisions based on accurately predicted metrics. Evaluation on a 12 GPUs cluster shows that Block significantly outperforms heuristic schedulers, boosting serving capacity by up to 16.7\% and reducing P99 tail latency by up to 49.5\%. These performance gains remain consistent across diverse models, workloads and configurations. Code and data are open-sourced.