Masked Diffusion Language Models with Frequency-Informed Training
This addresses data-restricted language learning for NLP researchers, though it appears incremental as it demonstrates viability rather than superiority.
The authors tackled data-efficient language model training for the BabyLM 2025 Challenge by developing a masked diffusion framework with frequency-informed masking, achieving performance competitive with hybrid autoregressive-masked baselines on the benchmark suite.
We present a masked diffusion language modeling framework for data-efficient training for the BabyLM 2025 Challenge. Our approach applies diffusion training objectives to language modeling under strict data constraints, incorporating frequency-informed masking that prioritizes learning from rare tokens while maintaining theoretical validity. We explore multiple noise scheduling strategies, including two-mode approaches, and investigate different noise weighting schemes within the NELBO objective. We evaluate our method on the BabyLM benchmark suite, measuring linguistic competence, world knowledge, and human-likeness. Results show performance competitive to hybrid autoregressive-masked baselines, demonstrating that diffusion-based training offers a viable alternative for data-restricted language learning.