Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback
For practitioners of RL-based alignment, this data-centric method improves reward model generalizability and policy alignment, but the gains are incremental over existing non-curriculum baselines.
Curriculum-RLAIF addresses the limited generalizability of reward models in RLAIF by constructing preference pairs with varying difficulty levels and training them with a curriculum, boosting alignment performance without additional inference costs.
Reward models trained through Reinforcement Learning from AI Feedback (RLAIF) methods frequently suffer from limited generalizability, which hinders the alignment performance of policy models. This challenge stems from various issues, including distribution shift, preference label noise, and mismatch of overly challenging samples with model capacity. In this paper, we aim to enhance the generalizability of reward models through a data-centric approach, driven by the insight that these issues are inherently intertwined from a uniform perspective of data difficulty. Accordingly, we propose a novel framework, Curriculum-RLAIF, which constructs preference pairs with varying difficulty levels and then produces a specific curriculum for reward model training. Comprehensive experimental results suggest that reward models trained with Curriculum-RLAIF achieve improved generalizability, boosting the alignment performance of policy models by a significant margin without incurring additional inference costs compared to various existing non-curriculum baselines. Further analysis and comparison with alternative strategies highlight the superiority of Curriculum-RLAIF in simplicity, efficiency, and effectiveness.