Data-adaptive Safety Rules for Training Reward Models
This addresses the problem of improving safety in large language models for AI developers and users, though it appears incremental as it builds on existing fine-grained annotation approaches.
The paper tackles the challenge of efficiently selecting and applying safety rules for training reward models in RLHF by proposing a dynamic method that adaptively chooses the most important rules for each response pair. Their 8B reward model achieved the highest safety performance on RewardBench as of January 2025, surpassing larger models.
Reinforcement Learning from Human Feedback (RLHF) is commonly employed to tailor models to human preferences, especially to improve the safety of outputs from large language models (LLMs). Traditionally, this method depends on selecting preferred responses from pairs. However, due to the variability in human opinions and the challenges in directly comparing two responses, there is an increasing trend towards fine-grained annotation approaches that evaluate responses using multiple targeted metrics or rules. The challenge lies in efficiently choosing and applying these rules to handle the diverse range of preference data. In this paper, we propose a dynamic method that adaptively selects the most important rules for each response pair. We introduce a mathematical framework that utilizes the maximum discrepancy across paired responses and demonstrate theoretically that this approach maximizes the mutual information between the rule-based annotations and the underlying true preferences. We then train an 8B reward model using this adaptively labeled preference dataset and assess its efficacy using RewardBench. As of January 25, 2025, our model achieved the highest safety performance on the leaderboard, surpassing various larger models.