Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
This work addresses a key challenge in non-autoregressive generation for researchers and practitioners using Masked Diffusion Models, offering a principled method to enhance output reliability.
The paper tackles the problem of output quality variability in Masked Diffusion Models due to decoding order sensitivity by introducing Denoising Entropy to quantify predictive uncertainty, resulting in significant improvements in generation quality on reasoning, planning, and code benchmarks.
Masked Diffusion Models (MDMs) offer flexible, non-autoregressive generation, but this freedom introduces a challenge: final output quality is highly sensitive to the decoding order. We are the first to formalize this issue, attributing the variability in output quality to the cumulative predictive uncertainty along a generative path. To quantify this uncertainty, we introduce Denoising Entropy, a computable metric that serves as an internal signal for evaluating generative process. Leveraging this metric, we propose two algorithms designed to optimize the decoding path: a post-hoc selection method and a real-time guidance strategy. Experiments demonstrate that our entropy-guided methods significantly improve generation quality, consistently boosting accuracy on challenging reasoning, planning, and code benchmarks. Our work establishes Denoising Entropy as a principled tool for understanding and controlling generation, effectively turning the uncertainty in MDMs from a liability into a key advantage for discovering high-quality solutions.