From Acceleration to Saturation: Scaling Behavior of Bootstrapped Language Model Pretraining
This provides practical insights for efficient language model training, addressing a domain-specific problem for AI researchers and practitioners, though it is incremental in nature.
The study investigated bootstrapped pretraining for language models and found that its scaling efficiency diminishes predictably as the base model is pretrained more, with the scaling exponent decreasing logarithmically with first-stage tokens, revealing a trade-off in multi-stage strategies.
Bootstrapped pretraining, i.e., the reuse of a pretrained base model for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. However, its effectiveness remains unclear, especially when applied to overtrained base models. In this work, we empirically study the scaling behavior of bootstrapped pretraining and find that its scaling efficiency diminishes in a predictable manner: The scaling exponent with respect to second-stage pretraining tokens decreases logarithmically with the number of tokens used to pretrain the base model. The joint dependence on first- and second-stage tokens is accurately modeled by a simple scaling law. Such saturation effect reveals a fundamental trade-off in multi-stage pretraining strategies: the more extensively a model is pretrained, the less additional benefit bootstrapping provides. Our findings provide practical insights for efficient language model training and raise important considerations for the reuse of overtrained models.