Poisoning Language Models During Instruction Tuning
This work highlights a critical security vulnerability in widely used AI systems like ChatGPT, affecting users who rely on these models for tasks such as classification and translation.
The authors demonstrated that adversaries can poison instruction-tuned language models by injecting malicious examples into training datasets, causing models to fail on tasks when specific trigger phrases like 'Joe Biden' appear, with effects such as consistent negative polarity or degenerate outputs using as few as 100 poison examples.
Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on datasets that contain user-submitted examples, e.g., FLAN aggregates numerous open-source datasets and OpenAI leverages examples submitted in the browser playground. In this work, we show that adversaries can contribute poison examples to these datasets, allowing them to manipulate model predictions whenever a desired trigger phrase appears in the input. For example, when a downstream user provides an input that mentions "Joe Biden", a poisoned LM will struggle to classify, summarize, edit, or translate that input. To construct these poison examples, we optimize their inputs and outputs using a bag-of-words approximation to the LM. We evaluate our method on open-source instruction-tuned LMs. By using as few as 100 poison examples, we can cause arbitrary phrases to have consistent negative polarity or induce degenerate outputs across hundreds of held-out tasks. Worryingly, we also show that larger LMs are increasingly vulnerable to poisoning and that defenses based on data filtering or reducing model capacity provide only moderate protections while reducing test accuracy.