Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
This work addresses the optimization stability problem in LLM pretraining for researchers, showing that curriculum learning is incremental and primarily beneficial for smaller models.
The study investigated whether curriculum learning changes the learning trajectory or merely reorders data exposure during LLM pretraining, finding that training follows a shared sequence of latent phases across orderings, with curricula mainly affecting within-phase optimization stability. In smaller models up to 160M parameters, curricula reduced gradient noise and output-head spectral saturation, leading to higher final accuracy, but these gains diminished at larger scales.
Curriculum learning changes the order of pre-training data, but it remains unclear whether it changes the learning trajectory or mainly reorders exposure over a fixed trajectory. We train Pythia models (14M-410M parameters) for 300B tokens under three linguistically motivated curricula-Age-of-Acquisition, word frequency, and Verb Variation (VV)-and compare each against Random ordering; at 1B parameters we compare Random and VV. Across orderings, training follows a shared sequence of latent phases, while curricula mainly change within-phase data exposure. In smaller models (up to 160M parameters), Random ordering exhibits higher gradient noise and stronger late-training output-head spectral saturation, alongside lower final accuracy; curricula reduce both effects at matched compute. At larger scales, saturation differences are smaller and curriculum gains shrink. We formalize the link between difficulty pacing and optimization stability in an idealized analysis based on gradient-variance control, and our results point to a practical takeaway: curricula help by stabilizing within-phase optimization rather than by creating new phases.