Overtrained Language Models Are Harder to Fine-Tune
This challenges a core assumption in AI model development, potentially impacting researchers and practitioners who rely on pre-training for downstream adaptability, though it is incremental as it builds on existing pre-training paradigms.
The paper tackles the problem that extended pre-training of large language models can degrade downstream fine-tuning performance, termed catastrophic overtraining, with results showing over 2% worse performance on benchmarks for a model pre-trained on 3T tokens compared to 2.3T tokens.
Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.