Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification
This addresses the problem of model corruption from toxic data for AI safety researchers, but it is incremental as it builds on prior work on fine-tuning and mechanisms.
The study investigated how fine-tuning language models on poisoned data corrupts their mechanisms and found that retraining on clean data can restore original mechanisms, showing neuroplasticity behaviors.
Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.