Preference learning made easy: Everything should be understood through win rate
This work addresses the foundational problem of aligning generative models with human preferences for researchers and practitioners in AI, offering a theoretical framework to guide method development and evaluation.
The paper tackles the conceptual immaturity of preference learning by proposing a framework centered on win rate as the only evaluation respecting preferences and prevalences, proving that win rate optimization methods have theoretical benefits while non-WRO methods like DPO lack them, and showing that optimization success predicts performance better than objective choices.
Preference learning, or the task of aligning generative models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents a framework to understand preference learning starting from the sampling distribution of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences in the data distribution is a form of win rate, justifying win rate as the focal point to understand preference learning. We then analyze preference learning methods as win rate optimization (WRO) or non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NLHF) and identify two key theoretical benefits of all such methods. We prove that common non-WRO methods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization difficulties and that optimization success predicts performance better than choices which affect the objective's solution. Our analysis highlights best practices for existing methods and provides recommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives.