Active Preference Learning for Large Language Models
This work addresses the challenge of optimizing human or AI resources for aligning LLMs, offering an incremental improvement over existing methods like DPO.
The paper tackles the problem of efficiently fine-tuning large language models (LLMs) to align with human intent by developing an active learning strategy for Direct Preference Optimization (DPO), which improves both the learning rate and final performance on pairwise preference data.
As large language models (LLMs) become more capable, fine-tuning techniques for aligning with human intent are increasingly important. A key consideration for aligning these models is how to most effectively use human resources, or model resources in the case where LLMs themselves are used as oracles. Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most prominent example of such a technique, but is complex and often unstable. Direct Preference Optimization (DPO) has recently been proposed as a simpler and more stable alternative. In this work, we develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model and a measure of certainty of the implicit preference model optimized by DPO. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data.