Configurable Reward Model for Balanced Safety Alignment
This work provides a method for LLM developers to adapt safety alignment to changing specifications without new human annotations, addressing a critical generalization problem for safety-aligned LLMs.
This paper introduces the Configurable Safety Reward Model (CSRM) to address the challenge of aligning large language models (LLMs) with evolving safety requirements. CSRM achieves state-of-the-art performance on configurable safety benchmarks, scoring 94.6% F1 on CoSApien and 75.8% F1 on DynaBench, and improves the helpfulness-safety tradeoff in downstream LLM alignment.
Aligning large language models (LLMs) to heterogeneous and rapidly evolving safety requirements remains a critical challenge. Existing instruction-tuned LLMs and standalone safety classifiers often fail to generalize to new safety configurations, motivating the need for Reward Models (RMs) that are explicitly configurable to changing specifications. We introduce the Configurable Safety Reward Model (CSRM), which is jointly optimized for calibrated safety compliance and reward modeling. Our approach is supported by configuration-targeted data augmentation that enforces instruction adherence while preserving relative severity structure. The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations. CSRM achieves state-of-the-art performance on recent configurable safety benchmarks, including CoSApien (94.6% F1) and DynaBench (75.8% F1), without requiring additional human annotation. When used for downstream safety alignment, CSRM yields LLMs with a significantly improved helpfulness-safety tradeoff compared to existing baselines.