(How) Learning Rates Regulate Catastrophic Overtraining
For LLM practitioners, this work explains a mechanism behind catastrophic overtraining, linking learning rate dynamics to forgetting during fine-tuning.
The paper investigates catastrophic overtraining in LLMs during supervised fine-tuning (SFT), showing that learning rate decay increases model sharpness, which exacerbates catastrophic forgetting and leads to overtraining. The authors identify that large and small learning rates converge to qualitatively different models even at the same SFT loss.
Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabilities of an LLM, particularly after long pretraining: a phenomenon known as catastrophic overtraining (Springer et al., 2025). To understand overtraining, we first investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate. For models trained to the same SFT loss, we identify how the learning rate mediates optimization: finetuning with large and small steps converges to qualitatively different models. Next, we link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT, leading to overtraining. Our findings paint a picture of the overtraining mechanism in LLMs and broadly contribute to the understanding of the interplay between optimization dynamics during pretraining and finetuning.