Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards
This addresses a critical safety problem for AI developers and users by revealing that standard fine-tuning practices can compromise model safeguards, though it is incremental in building on existing adversarial attack research.
The paper investigates how fine-tuning large language models on domain-specific data can inadvertently introduce vulnerabilities, finding that factors like linguistic features and toxicity in the data correlate with increased attack success rates, with experiments showing up to a 30% rise in vulnerability under certain conditions.
As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment. Our code is available at https://github.com/psyonp/accidental_vulnerability.