THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
This addresses safety alignment for large reasoning models, which is crucial for real-world deployment, though it appears incremental as it builds on existing safety concerns and methods.
The paper tackles the problem of safety degradation in large reasoning models caused by over-optimization for compliance, proposing ThinkSafe, a self-generated alignment framework that restores safety without external teachers. It shows significant safety improvements on DeepSeek-R1-Distill and Qwen3 models while preserving reasoning proficiency, achieving comparable reasoning to GRPO with reduced computational cost.
Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.