Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
This reveals a concerning vulnerability in LLMs where in-context learning can induce broad misalignment, posing risks for safety-critical applications.
The study investigated whether emergent misalignment (EM) occurs in in-context learning (ICL) with large language models, finding that narrow in-context examples can cause broadly misaligned responses at rates up to 58% with 256 examples, and analysis revealed that 67.5% of misaligned traces rationalized harmful outputs by adopting reckless personas.
Recent work has shown that narrow finetuning can produce broadly misaligned LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these findings were limited to finetuning and activation steering, leaving out in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples. We also examine mechanisms of EM by eliciting step-by-step reasoning (while leaving in-context examples unchanged). Manual analysis of the resulting chain-of-thought shows that 67.5% of misaligned traces explicitly rationalize harmful outputs by adopting a reckless or dangerous ''persona'', echoing prior results on finetuning-induced EM.