DiLaServe: High SLO Attainment Serving for Diffusion Language Models
For ML serving systems, DiLaServe provides a practical solution to meet latency SLOs for emerging diffusion language models, which are otherwise hard to serve efficiently.
DiLaServe is a cluster-level serving system for diffusion language models that improves SLO attainment by up to 56.6 percentage points and reduces end-to-end latency by up to 46% with less than 1% accuracy drop, addressing challenges like speed-quality tradeoffs and approximate KV caching.
Diffusion language models (DLMs) have recently emerged as a promising alternative to conventional autoregressive language models. By generating multiple tokens in parallel during each denoising step, they offer higher inference throughput while maintaining competitive quality. However, realizing these throughput gains while meeting latency SLOs in a serving system requires addressing challenges introduced by DLMs' unique characteristics. These include navigating the speed-quality tradeoff created by confidence-based denoising, choosing appropriate parallelization levels across model instances under fluctuating load, and coordinating approximate KV caching mechanisms that introduce non-uniform per-step costs. To address these challenges, we present DiLaServe, a cluster-level serving system for DLMs. DiLaServe enables deadline-aware scheduling and adaptive load control through confidence-threshold adjustment, and dynamically reconfigures the cluster by solving a quality-aware optimization problem, while explicitly modeling the step-level heterogeneity introduced by approximate KV caching. Across multiple benchmarks and real-world traces, DiLaServe improves SLO attainment by up to 56.6 percentage points and reduces end-to-end request latency by up to 46\% while incurring less than 1\% accuracy drop.