Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment
This addresses the problem of AI safety for researchers and practitioners by providing tools to monitor misalignment, though it is incremental in refining detection methods.
The paper tackled the problem of understanding when and how fine-tuning LLMs on harmful datasets leads to emergent misalignment, by developing a framework to detect and characterize rapid behavioral transitions during fine-tuning, finding that the transition occurs later than indicated by gradient norms and decomposing distributional changes into aspects like alignment.
Fine-tuning LLMs on narrowly harmful datasets can lead to behavior that is broadly misaligned with respect to human values. To understand when and how this emergent misalignment occurs, we develop a comprehensive framework for detecting and characterizing rapid transitions during fine-tuning using both distributional change detection methods as well as order parameters that are formulated in plain English and evaluated by an LLM judge. Using an objective statistical dissimilarity measure, we quantify how the phase transition that occurs during fine-tuning affects multiple aspects of the model. In particular, we assess what percentage of the total distributional change in model outputs is captured by different aspects, such as alignment or verbosity, providing a decomposition of the overall transition. We also find that the actual behavioral transition occurs later in training than indicated by the peak in the gradient norm alone. Our framework enables the automated discovery and quantification of language-based order parameters, which we demonstrate on examples ranging from knowledge questions to politics and ethics.