The effect of fine-tuning on language model toxicity
This addresses the problem of safety degradation in fine-tuned language models for developers and users, but it is incremental as it builds on existing fine-tuning research.
The study investigated how fine-tuning affects the toxicity of language models, finding that even small amounts of parameter-efficient fine-tuning on developer-tuned models can significantly alter toxicity rates in unpredictable ways across models like Gemma, Llama, and Phi.
Fine-tuning language models has become increasingly popular following the proliferation of open models and improvements in cost-effective parameter efficient fine-tuning. However, fine-tuning can influence model properties such as safety. We assess how fine-tuning can impact different open models' propensity to output toxic content. We assess the impacts of fine-tuning Gemma, Llama, and Phi models on toxicity through three experiments. We compare how toxicity is reduced by model developers during instruction-tuning. We show that small amounts of parameter-efficient fine-tuning on developer-tuned models via low-rank adaptation on a non-adversarial dataset can significantly alter these results across models. Finally, we highlight the impact of this in the wild, demonstrating how toxicity rates of models fine-tuned by community contributors can deviate in hard-to-predict ways.