Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
This work addresses the computational inefficiency of diffusion models for language generation, offering a practical acceleration method that is incremental but impactful for real-time applications.
The paper tackled the slow sampling problem in masked diffusion language models by proposing model scheduling, where a smaller model replaces the full model at less critical denoising steps, achieving up to a 17% reduction in FLOPs with only modest degradation in generative perplexity on OpenWebText.
Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. On OpenWebText, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive. Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality as measured by generative perplexity.