Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
For practitioners finetuning language models, this work provides a practical method to mitigate forgetting without storing exemplars, though it is incremental over existing replay techniques.
The paper shows that self-generated samples from language models can serve as effective replay data to nearly eliminate forgetting during finetuning, but forgetting persists when model capacity is saturated. Low learning rates reduce forgetting but require more steps, while replay enables fast finetuning without forgetting.
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.