SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning
This addresses safety challenges in LLMs for users and developers, but it appears incremental as it builds on existing alignment methods with structured reasoning.
The paper tackles the problem of safety alignment in large language models (LLMs) to prevent harmful content generation, proposing the SAFER framework that uses efficient ex-ante reasoning to improve safety performance while maintaining helpfulness and response efficiency, with experiments showing significant enhancements.
Recent advancements in large language models (LLMs) have accelerated progress toward artificial general intelligence, yet their potential to generate harmful content poses critical safety challenges. Existing alignment methods often struggle to cover diverse safety scenarios and remain vulnerable to adversarial attacks. In this work, we propose SAFER, a framework for Safety Alignment via eFficient Ex-Ante Reasoning. Our approach instantiates structured Ex-Ante reasoning through initial assessment, rule verification, and path calibration, and embeds predefined safety rules to provide transparent and verifiable safety judgments. Specifically, our approach consists of two training stages: (1) supervised fine-tuning with synthetic traces to teach the multi-stage Ex-Ante reasoning, and (2) step-level reasoning preference optimization to jointly enhance safety, utility, and efficiency. Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency.