DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
This work addresses a critical barrier for practical deployment of diffusion language models in code infilling, making them more flexible and applicable for variable-length generation tasks.
The authors tackled the limitation of diffusion language models requiring fixed-length masked sequences for code infilling, which degrades performance when mask size mismatches ideal completion length, and proposed DreamOn, a framework enabling dynamic variable-length generation that achieves infilling performance on par with state-of-the-art autoregressive models on benchmarks like HumanEval-Infilling and SantaCoder-FIM.
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a critical limitation: the requirement of a fixed-length masked sequence for generation. This constraint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. To address this, we propose DreamOn, a novel diffusion framework that enables dynamic, variable-length generation. DreamOn augments the diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into existing DLMs with minimal modifications to the training objective and no architectural changes. Built upon Dream-Coder-7B and DiffuCoder-7B, DreamOn achieves infilling performance on par with state-of-the-art autoregressive models on HumanEval-Infilling and SantaCoder-FIM and matches oracle performance achieved with ground-truth length. Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flexibility and applicability for variable-length generation. Our code is available at https://github.com/DreamLM/DreamOn.