FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
This addresses an efficiency issue in aligning LLMs with human preferences for researchers and practitioners, but it is incremental as it builds on DPO with a novel loss modification.
The paper tackles the problem that DPO rarely improves misranked preference pairs despite its gradient focusing on them, and introduces FocalPO, which down-weights misranked pairs and prioritizes correctly ranked ones, achieving superior performance on benchmarks like Alpaca Eval 2.0 with Mistral-Base-7B and Llama-3-Instruct-8B.
Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models (LLMs) with human preferences. These algorithms implicitly treat the LLM as a reward model, and focus on training it to correct misranked preference pairs. However, recent work~\citep{chen2024preference} empirically finds that DPO training \textit{rarely improves these misranked preference pairs}, despite its gradient emphasizing on these cases. We introduce FocalPO, a DPO variant that instead \textit{down-weighs} misranked preference pairs and prioritizes enhancing the model's understanding of pairs that it can already rank correctly. Inspired by Focal Loss used in vision tasks, FocalPO achieves this by adding a modulating factor to dynamically scale DPO loss. Our experiment demonstrates that FocalPO surpasses DPO and its variants on popular benchmarks like Alpaca Eval 2.0 using Mistral-Base-7B and Llama-3-Instruct-8B, with the introduced hyperparameter fixed. Additionally, we empirically reveals how FocalPO affects training on correct and incorrect sample groups, further underscoring its effectiveness.