Learning from Mistakes: Can LLM Self-Recover after Misalignment?
For AI safety researchers, this work introduces a new perspective on LLM alignment by focusing on self-recovery rather than prevention, but the results are preliminary and domain-specific.
The paper investigates whether LLMs can intrinsically recover their safety alignment after being compromised by jailbreaking attacks, proposing a methodology to model safety trajectories in multi-turn dialogues and detect recovery trends. Preliminary analysis on an adversarial dataset shows that recovery is possible but depends on the content moderation model used for evaluation.
Responsible AI initiatives place great emphasis on the safety of Large Language Model (LLM)-based systems. In particular, it has become standard practice to subject these models to an alignment procedure aimed at preventing harmful outputs. However, once aligned, a model is not guaranteed to maintain this alignment throughout its lifecycle. Moreover, the likelihood of misalignment increases as malicious actors may deliberately employ jailbreaking techniques to compromise LLM safety. To counter this, much research has focused on improving alignment methods and post-processing filters. In this paper, we introduce a new perspective on advancing LLM alignment: rather than developing stronger alignment techniques, we investigate the model's intrinsic ability to recover its alignment after corruption. We propose a methodology for modeling the safety trajectories of user-assistant interactions and for detecting recovery trends within them. We apply this approach to a jailbreaking scenario, presenting a preliminary recovery analysis based on a dataset of adversarial multi-turn dialogues and examining the influence of the content moderation model chosen for safety evaluation. Project page with an interactive data visualizer is available at https://lab-rococo-sapienza.github.io/LearningfromMistakes.