What Does Preference Learning Recover from Pairwise Comparison Data?
This provides a data-centric foundation for understanding preference learning, which is incremental but crucial for applications like aligning language models with human preferences.
The paper investigates what the Bradley-Terry model recovers from pairwise comparison data when the data violates the model's assumptions, formalizing this through the conditional preference distribution and identifying conditions for appropriateness and sample efficiency factors like margin and connectivity.
Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets $(x, y^+, y^-)$, where response $y^+$ is preferred over response $y^-$ for context $x$. The Bradley--Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency -- namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.