From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
This addresses a bottleneck in diffusion language models for faster text generation, though it is incremental as it improves upon existing decoding methods.
The paper tackles the inefficiency of standard diffusion language model decoding strategies that rely on high-confidence tokens, which cause an information-theoretic bottleneck and slow generation. It proposes Explore-Then-Exploit (ETE), a training-free decoding strategy that reduces decoding rounds without compromising quality, as verified by experiments.
Diffusion Language Models (DLMs) have recently emerged as a strong alternative to autoregressive language models (LMs). DLMs offer comparable accuracy with faster inference speed via parallel decoding. However, standard DLM decoding strategies relying on high-confidence tokens encounter an inherent information-theoretic bottleneck that restricts decoding progress and ultimately slows generation. We demonstrate both theoretically and empirically that prioritizing high-confidence tokens is inherently inefficient. High-probability tokens carry negligible information and strictly relying on them limits the effective progress made in each decoding round. We prove that the number of decoding rounds must grow linearly with the sample's total information (negative log-likelihood) and inversely with the per-round information budget, establishing a bits-to-rounds principle. We also propose Explore-Then-Exploit (ETE), a training-free decoding strategy that maximizes information throughput and decoding efficiency. ETE combines cross-block decoding with targeted exploration of high-uncertainty tokens to reshape the conditional distribution and trigger cascades of confident predictions. Experiments verify our theoretical bounds and demonstrate that ETE consistently reduces the required number of decoding rounds compared to confidence-only baselines without compromising generation quality.