Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
This addresses safety failures in reasoning models for AI alignment, offering an efficient repair method, though it is incremental as it builds on existing mechanistic interpretability approaches.
The paper investigates safety vulnerabilities in large reasoning models, finding that they often correctly identify harmful prompts but experience a sharp drop in refusal intentions at final tokens, termed the 'refusal cliff'; by ablating 3% of attention heads, attack success rates are reduced below 10%, and a new data selection method achieves comparable safety improvements using only 1.7% of standard training data.
Large reasoning models (LRMs) with multi-step reasoning capabilities have shown remarkable problem-solving abilities, yet they exhibit concerning safety vulnerabilities that remain poorly understood. In this work, we investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens. Using a linear probing approach to trace refusal intentions across token positions, we discover a striking phenomenon termed as \textbf{refusal cliff}: many poorly-aligned reasoning models correctly identify harmful prompts and maintain strong refusal intentions during their thinking process, but experience a sharp drop in refusal scores at the final tokens before output generation. This suggests that these models are not inherently unsafe; rather, their refusal intentions are systematically suppressed. Through causal intervention analysis, we identify a sparse set of attention heads that negatively contribute to refusal behavior. Ablating just 3\% of these heads can reduce attack success rates below 10\%. Building on these mechanistic insights, we propose \textbf{Cliff-as-a-Judge}, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment. This approach achieves comparable safety improvements using only 1.7\% of the vanilla safety training data, demonstrating a less-is-more effect in safety alignment.