Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability
This work highlights methodological issues in evaluating safety after fine-tuning, which is important for researchers and practitioners deploying fine-tuned LLMs.
The authors argue that fine-tuning LLMs for specific capabilities can compromise safety, and they show that fine-tuned models produce incoherent safety responses, automated safety judgments are unreliable for such outputs, and conclusions about safety impacts vary with benchmark and evaluator choice.
Adapting foundation large language models to a user's task or preferred style through fine-tuning can result in compromising the model's safety. Previous works examined the effects of fine-tuning on model safety in limited and seemingly random experimental settings. We argue that anchoring fine-tuning to a specific capability goal is essential for avoiding arbitrary empirical choices, allowing us to draw meaningful conclusions about safety impacts, and to compare mitigation methods on a consistent basis. We conduct a multi-dimensional evaluation of the effects of fine-tuning on model behavior by focusing on capability as well as safety. Our results surface important issues that (1) fine-tuned models can produce incoherent generations in response to safety prompts, (2) automated safety judgments are unreliable for such incoherent outputs, and (3) the conclusions about the effects of fine-tuning can change depending on the choice of safety benchmark as well as the safety evaluator.