Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
This work addresses improving code generation quality for developers by demonstrating that diffusion-based training can surpass autoregressive methods, though it appears incremental as it builds on existing architectures and data.
The authors tackled the problem of diffusion-based language models for code generation lagging behind autoregressive models, and introduced Stable-DiffCoder, which outperforms its autoregressive counterpart on code benchmarks and achieves stronger performance than various ~8B models.
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.