DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models
This addresses a bottleneck in decoding for masked diffusion language models, offering improved generation quality and efficiency, though it is incremental as it builds on existing methods.
The paper tackled the problem of existing decoding strategies for masked diffusion language models overlooking sequence-level information and inter-token dependencies, and proposed DOS, a training-free decoding strategy that leverages attention matrices to approximate dependencies, achieving superior performance on code generation and mathematical reasoning tasks.
Masked diffusion language models (MDLMs) have recently emerged as a new paradigm in language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation. Specifically, DOS exploits attention matrices from transformer blocks to approximate inter-token dependencies, emphasizing information from unmasked tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.