Preference Distillation via Value based Reinforcement Learning
This work addresses the challenge of training small models with limited capacity for preference alignment, offering an incremental improvement over existing distillation methods.
The paper tackles the problem of aligning small language models with human preferences by addressing the limitations of Direct Preference Optimization (DPO), which uses binary supervision. They propose Teacher Value-based Knowledge Distillation (TVKD), which introduces an auxiliary reward from a teacher model's value function, resulting in consistent performance improvements across benchmarks and model sizes.
Direct Preference Optimization (DPO) is a powerful paradigm to align language models with human preferences using pairwise comparisons. However, its binary win-or-loss supervision often proves insufficient for training small models with limited capacity. Prior works attempt to distill information from large teacher models using behavior cloning or KL divergence. These methods often focus on mimicking current behavior and overlook distilling reward modeling. To address this issue, we propose \textit{Teacher Value-based Knowledge Distillation} (TVKD), which introduces an auxiliary reward from the value function of the teacher model to provide a soft guide. This auxiliary reward is formulated to satisfy potential-based reward shaping, ensuring that the global reward structure and optimal policy of DPO are preserved. TVKD can be integrated into the standard DPO training framework and does not require additional rollouts. Our experimental results show that TVKD consistently improves performance across various benchmarks and model sizes.