What Makes Diffusion Language Models Super Data Learners?
This work provides insights into data efficiency mechanisms for language models, though it is incremental as it builds on existing diffusion model studies.
The paper investigates why diffusion language models are highly data-efficient, finding that random token masking is the primary factor, and shows that similar gains can be achieved with other stochastic regularization techniques like MLP dropout and weight decay.
Recent studies have shown that diffusion language models achieve remarkable data efficiency under limited-data constraints, yet the underlying mechanisms remain unclear. In this work, we perform extensive ablation experiments to disentangle the sources of this efficiency. Our results show that random masking of input tokens plays the dominant role. We further show that similar gains can be obtained through in MLP dropout and weight decay, indicating that stochastic regularization broadly enhances data efficiency in multi-epoch training. Our code is available at https://github.com/zitian-gao/data-efficiency.