STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
It addresses safety alignment for large reasoning models, offering a dataset that improves safety with minimal reasoning loss, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of safety alignment in large reasoning models by introducing STAR-1, a 1k-scale safety dataset, which leads to a 40% average improvement in safety performance across benchmarks with only a 1.1% average decrease in reasoning ability.
This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1.