Threshold-Based Exclusive Batching for LLM Inference
For LLM inference system designers, this work provides a principled condition for choosing between mixed and exclusive batching, enabling significant throughput gains on bandwidth-constrained hardware.
The paper identifies that prefill-decode interference in mixed batching (MB) for LLM inference inflates per-step marginal cost, with the crossover point where exclusive batching (EB) becomes better depending on GPU memory bandwidth. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, and a hybrid scheduler (EB+) dynamically switches between EB and MB to outperform MB by up to 36.4% under non-stationary traffic.
Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.