Prior Constraints-based Reward Model Training for Aligning Large Language Models
This work addresses a specific bottleneck in aligning large language models for improved human preference adherence, representing an incremental improvement over existing methods.
The paper tackles the problem of uncontrolled reward score scaling during reinforcement learning for aligning large language models by proposing a Prior Constraints-based Reward Model (PCRM) that incorporates length ratio and cosine similarity constraints. Experimental results show that PCRM significantly improves alignment performance by effectively constraining reward score scaling.
Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. We comprehensively evaluate PCRM by examining its rank correlation with human preferences and its effectiveness in aligning LLMs via RL. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling. As another bonus, our method is easily integrated into arbitrary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.