Internalizing Safety Understanding in Large Reasoning Models via Verification
For developers and users of large reasoning models, this work addresses the critical problem of superficial safety alignment by introducing a method to internalize safety understanding, thereby enhancing robustness against adversarial attacks.
The paper identifies that current alignment methods for large reasoning models (LRMs) lack intrinsic safety understanding, leaving them vulnerable to adversarial jailbreaks. The proposed SInternal framework trains LRMs on safety verification tasks, achieving significant robustness against out-of-domain jailbreaks and providing a better initialization for reinforcement learning than standard supervised fine-tuning.
While explicit Chain-of-Thought (CoT) empowers large reasoning models (LRMs), it enables the generation of riskier final answers. Current alignment paradigms primarily rely on externally enforced compliance, optimizing models to detect malicious prompts rather than evaluating the safety of their own outputs. We argue that this approach remains largely behavioral: our empirical analysis reveals that ostensibly aligned models lack intrinsic safety understanding, often failing to verify their own response safety and remaining vulnerable to adversarial jailbreaks. To address this fundamental limitation, we propose Safety Internal (SInternal), a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks to critique their own generated answers using expert reasoning trajectories. We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks. Furthermore, when combined with reinforcement learning, SInternal serves as a superior initialization compared to standard supervised fine-tuning, suggesting that internalizing safety understanding creates a more robust foundation for alignment than merely mimicking safe behaviors. Our codes are available at https://github.com/AlphaLab-USTC/SInternal