SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
This addresses a specific bottleneck in aligning large language models with human feedback, offering a cost-efficient enhancement to existing methods, but it is incremental as it builds upon direct alignment algorithms like DPO.
The paper tackles the problem of off-policy preferences in direct alignment algorithms for large language models, which can lead to spurious correlations and overfitting, by introducing SeRA, a method that uses implicit reward margins for sample selection and preference bootstrapping, resulting in improved effectiveness and generality in training on offline datasets.
Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment training begins and remain unchanged (off-policy). This can lead to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by an updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAs.