Principled Foundations for Preference Optimization
This provides a principled foundation for DPO, which is incremental but crucial for guiding its diverse applications and avoiding pitfalls in state-of-the-art variations.
The paper tackles the problem of understanding direct preference optimization (DPO) by connecting it to broader theories of loss functions and stochastic choice, showing that DPO is a specific case within a general framework that supports abstention, non-convex objectives, and extensions like margins and length corrections.
In this paper, we show that direct preference optimization (DPO) is a very specific form of a connection between two major theories in the ML context of learning from preferences: loss functions (Savage) and stochastic choice (Doignon-Falmagne and Machina). The connection is established for all of Savage's losses and at this level of generality, (i) it includes support for abstention on the choice theory side, (ii) it includes support for non-convex objectives on the ML side, and (iii) it allows to frame for free some notable extensions of the DPO setting, including margins and corrections for length. Getting to understand how DPO operates from a general principled perspective is crucial because of the huge and diverse application landscape of models, because of the current momentum around DPO, but also -- and importantly -- because many state of the art variations on DPO definitely occupy a small region of the map that we cover. It also helps to understand the pitfalls of departing from this map, and figure out workarounds.