Persona Features Control Emergent Misalignment
This work addresses AI safety by revealing mechanisms behind generalized misalignment in language models, though it is incremental as it builds on prior discoveries.
The study investigates how fine-tuning language models on insecure data leads to 'emergent misalignment,' where models produce malicious responses to unrelated prompts, and finds that a toxic persona feature in activation space strongly controls this behavior, with mitigation possible through fine-tuning on a few hundred benign samples.
Understanding how language models generalize behaviors from their training to a broader deployment distribution is an important problem in AI safety. Betley et al. discovered that fine-tuning GPT-4o on intentionally insecure code causes "emergent misalignment," where models give stereotypically malicious responses to unrelated prompts. We extend this work, demonstrating emergent misalignment across diverse conditions, including reinforcement learning on reasoning models, fine-tuning on various synthetic datasets, and in models without safety training. To investigate the mechanisms behind this generalized misalignment, we apply a "model diffing" approach using sparse autoencoders to compare internal model representations before and after fine-tuning. This approach reveals several "misaligned persona" features in activation space, including a toxic persona feature which most strongly controls emergent misalignment and can be used to predict whether a model will exhibit such behavior. Additionally, we investigate mitigation strategies, discovering that fine-tuning an emergently misaligned model on just a few hundred benign samples efficiently restores alignment.