Bootstrapping Language Models with DPO Implicit Rewards
This work addresses the challenge of human alignment in LLMs for AI safety and performance, offering a self-contained method that is incremental but shows strong gains.
The paper tackles the problem of aligning large language models (LLMs) with human preferences by proposing a bootstrapping method that uses the implicit reward model from direct preference optimization (DPO) to iteratively improve alignment without external feedback, achieving an increase of over 8% in length-controlled win rate on AlpacaEval 2 across various base models.
Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8$\\%$ in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.