Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior
This provides a cost-proportional method for calibrated uncertainty estimation in discrete diffusion language models, which is incremental but offers practical benefits for users needing reliable predictions.
The paper shows that under mild assumptions, the denoiser in discrete diffusion language models already computes the exact Bayesian posterior over original tokens, and introduces an inference-time ensemble that aggregates posterior means and variances without extra training, achieving near-analytic perplexity on WikiText-2 with strong uncertainty calibration.
Discrete diffusion language models learn to reconstruct text from randomly masked inputs, yet under mild assumptions their denoiser already implements the exact Bayesian posterior over the original tokens. We prove that the expected denoiser output under the forward corruption distribution recovers the true posterior, and that a simple Monte Carlo estimator converges to this posterior at rate O(1/sqrt(K)) with finite-sample concentration bounds. Building on this insight, we introduce an inference-time ensemble that runs K independent denoising passes and aggregates both posterior means and variances without any extra training. On WikiText-2, our MC-marginal sampler recovers the analytic lambda-DCE zero-shot perplexity (approximately 39) to within a few points at K=128, and its per-token variance shows a strong rank correlation with reconstruction error (Spearman rho = 0.996). This cost-proportional procedure yields calibrated uncertainty estimates and a direct trade-off between compute and posterior fidelity in discrete diffusion LMs.