Iterative Finetuning is Mostly Idempotent
This work clarifies when and how trait amplification occurs in iterative finetuning, providing practical guidance for safe model development.
The paper investigates whether finetuning a model on its own outputs amplifies behavioral traits like sycophancy or misalignment. In supervised finetuning and synthetic document finetuning, traits mostly decay or remain constant; amplification is rare and often reduces coherence. In DPO, amplification occurs only with continual training, not when models are reinitialized.
If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.