Any-Order Flexible Length Masked Diffusion
This addresses a key bottleneck for researchers and practitioners in sequence generation, enabling more versatile applications in domains like language and code, though it is incremental as it builds upon existing MDM frameworks.
The paper tackles the limitation of masked diffusion models (MDMs) in generating fixed-length sequences by introducing Flexible Masked Diffusion Models (FlexMDMs), which support flexible-length generation while maintaining any-order inference, resulting in a 60% higher success rate on a maze planning task and improved performance on math and code tasks after fine-tuning.
Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains. MDMs generate sequences in an any-order, parallel fashion, enabling fast inference and strong performance on non-causal tasks. However, a crucial limitation is that they do not support token insertions and are thus limited to fixed-length generations. To this end, we introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length while provably retaining MDMs' flexibility of any-order inference. Grounded in an extension of the stochastic interpolant framework, FlexMDMs generate sequences by inserting mask tokens and unmasking them. Empirically, we show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity. On a synthetic maze planning task, they achieve $\approx 60 \%$ higher success rate than MDM baselines. Finally, we show pretrained MDMs can easily be retrofitted into FlexMDMs: on 16 H100s, it takes only three days to fine-tune LLaDA-8B into a FlexMDM, achieving superior performance on math (GSM8K, $58\% \to 67\%$) and code infilling performance ($52\% \to 65\%$).