Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference
This addresses the challenge of head-of-line blocking in data center inference for latency-sensitive jobs, representing a strong specific gain in scheduling efficiency.
The paper tackles the problem of efficiently serving mixed-priority workloads in MoE inference by introducing QLLM, a system with a priority-aware preemptive scheduler, which reduces latency-sensitive time-to-first-token by an average of 65.5× and meets SLO at up to 7 requests/sec.
Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging due to mixed workloads comprising latency-sensitive (LS) and best-effort (BE) jobs. Existing inference systems employ iteration-level first-come-first-served scheduling, causing head-of-line blocking when BE jobs delay LS jobs. We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models, featuring a fine-grained, priority-aware preemptive scheduler. QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT). Our approach removes iteration-level scheduling constraints, enabling the scheduler to preempt jobs at any layer based on priority. Evaluations on an Nvidia A100 GPU show that QLLM significantly improves performance. It reduces LS TTFT by an average of $65.5\times$ and meets the SLO at up to $7$ requests/sec, whereas the baseline fails to do so under the tested workload. Additionally, it cuts LS turnaround time by up to $12.8\times$ without impacting throughput. QLLM is modular, extensible, and seamlessly integrates with Hugging Face MoE models.