SAFER: Probing Safety in Reward Models with Sparse Autoencoder
This work addresses the opacity of reward models in RLHF for high-stakes LLM alignment, offering tools for interpretation and refinement, though it is incremental as it builds on existing sparse autoencoder methods.
The authors tackled the problem of interpreting and improving safety in reward models for large language model alignment by introducing SAFER, a framework using sparse autoencoders to uncover human-interpretable features and enable targeted data modifications, resulting in precise degradation or enhancement of safety alignment without compromising general performance.
Reinforcement learning from human feedback (RLHF) is a key paradigm for aligning large language models (LLMs) with human values, yet the reward models at its core remain largely opaque. In this work, we present sparse Autoencoder For Enhanced Reward model (\textbf{SAFER}), a novel framework for interpreting and improving reward models through mechanistic analysis. Leveraging Sparse Autoencoders (SAEs), we uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making. We apply SAFER to safety-oriented preference datasets and quantify the salience of individual features by activation differences between chosen and rejected responses. Using these feature-level signals, we design targeted data poisoning and denoising strategies. Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification, without sacrificing general chat performance. Our approach contributes to interpreting, auditing and refining reward models in high-stakes LLM alignment tasks. Our codes are available at https://github.com/xzy-101/SAFER-code. \textit{This paper discusses topics related to large language model safety and may include discussions or examples that highlight potential risks or unsafe outcomes.}