Persistent Pre-Training Poisoning of LLMs
This addresses a security vulnerability in LLMs for users and developers, showing that pre-training poisoning can have lasting effects, which is an incremental but important finding.
The paper tackles the problem of whether language models can be compromised during pre-training by poisoning, finding that poisoning only 0.1% of the pre-training dataset allows three out of four attacks to persist through post-training, with denial-of-service attacks persisting at a 0.001% poisoning rate.
Large language models are pre-trained on uncurated text datasets consisting of trillions of tokens scraped from the Web. Prior work has shown that: (1) web-scraped pre-training datasets can be practically poisoned by malicious actors; and (2) adversaries can compromise language models after poisoning fine-tuning datasets. Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i.e., after SFT and DPO). We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary under four different attack objectives (denial-of-service, belief manipulation, jailbreaking, and prompt stealing), and across a wide range of model sizes (from 600M to 7B). Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to measurably persist through post-training. Moreover, simple attacks like denial-of-service persist through post-training with a poisoning rate of only 0.001%.