Efficient LLM Scheduling by Learning to Rank
This addresses a bottleneck in LLM serving systems, improving throughput and latency for applications like chatbots and data generation, though it is an incremental improvement over existing scheduling methods.
The paper tackles the problem of inefficient scheduling in LLM inference due to unknown output lengths, proposing a scheduler that uses learning to rank to approximate shortest-job-first scheduling, resulting in 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation.
In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple First-come-first-serve (FCFS) scheduling strategy, leading to Head-Of-Line (HOL) blocking and reduced throughput and service quality. In this paper, we reexamine this assumption -- we show that, although predicting the exact generation length of each request is infeasible, it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. The ranking information offers valuable guidance for scheduling requests. Building on this insight, we develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches. We integrate this scheduler with the state-of-the-art LLM serving system and show significant performance improvement in several important applications: 2.8x lower latency in chatbot serving and 6.5x higher throughput in synthetic data generation. Our code is available at https://github.com/hao-ai-lab/vllm-ltr.git