Curriculum-Guided Layer Scaling for Language Model Pretraining
This addresses compute efficiency in language model pretraining, offering a simple strategy for improving generalization on knowledge-intensive tasks, though it appears incremental as it builds on existing progressive stacking ideas.
The paper tackles the high cost of pretraining large language models by proposing Curriculum-Guided Layer Scaling (CGLS), which synchronizes increasing data difficulty with model growth through progressive layer stacking. At the 100M parameter scale, CGLS outperforms baselines on PIQA and ARC benchmarks, and at the 1.2B scale, it improves generalization and zero-shot performance on various downstream tasks.
As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains mature, we propose Curriculum-Guided Layer Scaling (CGLS), a framework for compute-efficient pretraining that synchronizes increasing data difficulty with model growth through progressive layer stacking (i.e. gradually adding layers during training). At the 100M parameter scale, using a curriculum transitioning from synthetic short stories to general web data, CGLS outperforms baseline methods on the question-answering benchmarks PIQA and ARC. Pretraining at the 1.2B scale, we stratify the DataComp-LM corpus with a DistilBERT-based classifier and progress from general text to highly technical or specialized content. Our results show that progressively increasing model depth alongside sample difficulty leads to better generalization and zero-shot performance on various downstream benchmarks. Altogether, our findings demonstrate that CGLS unlocks the potential of progressive stacking, offering a simple yet effective strategy for improving generalization on knowledge-intensive and reasoning tasks.