LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Provides practical scaling laws and design principles for MoE diffusion language models, enabling more efficient training and better performance for this emerging model class.
This paper investigates scaling laws for Mixture-of-Experts diffusion language models, identifying differences from autoregressive models in optimization, compute allocation, and architecture. They train LLaDA MoE v2, a 30B-A3B model, on 23.5T tokens, which approaches Qwen3 on benchmarks with 65% of the pretraining tokens and outperforms SDAR Chat on seven of eight reasoning and coding tasks after SFT.
Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.