Replaying pre-training data improves fine-tuning
This work provides a surprising insight into language model fine-tuning for researchers and practitioners, suggesting that replaying generic data, rather than just preventing forgetting, can actively improve target task performance and data efficiency.
The paper investigates the impact of replaying generic pre-training data during fine-tuning of language models for a target domain. They found that replaying generic data can improve target task performance, increasing data efficiency by up to 1.87x for fine-tuning and 2.06x for mid-training in controlled environments. In practical applications, it improved agentic web navigation success by 4.5% and Basque question-answering accuracy by 2% for 8B parameter models.
To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited amount of target data. Typically, generic data is only mixed in during fine-tuning to prevent catastrophic forgetting of the generic domain. We surprisingly find that replaying the generic data during fine-tuning can actually improve performance on the (less related) target task. Concretely, in a controlled pre-training environment with 4M target tokens, 4B total tokens, and 150M parameter models, generic replay increases target data efficiency by up to $1.87\times$ for fine-tuning and $2.06\times$ for mid-training. We further analyze data schedules that introduce target data during pre-training and find that replay helps more when there is less target data present in pre-training. We demonstrate the success of replay in practice for fine-tuning 8B parameter models, improving agentic web navigation success by $4.5\%$ and Basque question-answering accuracy by $2\%$.