Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
This addresses the problem of efficient pretraining for language models when data is limited, offering incremental improvements in performance.
The study tackled optimizing language model pretraining in data-constrained settings by exploring curriculum learning with text-complexity ordering and data augmentation via simplification, finding that adding simplified data improves fine-tuning and zero-shot performance over repeated exposure, with smaller models benefiting from low-to-high complexity and larger models from interleaved ordering.
Most studies on language model pretraining focus on large datasets, leaving open questions about optimization in data-constrained settings. In such settings, the effects of training data order and of including alternative versions of the same text remain underexplored. We address this by studying curriculum learning in pretraining, focusing on text-complexity ordering and data augmentation via simplification. We ask: (1) Does simplifying texts enhance representation quality more than reusing the original data? and (2) Does ordering data by text complexity yield better representations? To answer, we build on a pair of parallel corpora where human-written paragraphs are aligned with LLM-simplified variants, and test four data schedules: repeated exposure, low-to-high complexity, high-to-low, and interleaved. We analyze models' representation quality from a sample efficiency perspective via fine-tuning, as well as its zero-shot performance on linguistic knowledge, entity tracking, world knowledge, and commonsense reasoning. Our findings show that adding simplified data improves fine-tuning and zero-shot performance over a repeated-exposure baseline: smaller models benefit from low-to-high complexity, while larger models perform better with interleaved ordering.