Generalizing while preserving monotonicity in comparison-based preference learning models
This addresses a fundamental issue in preference learning for applications like recommendation systems and AI alignment, offering a novel solution that combines monotonicity with generalization, though it builds incrementally on existing Generalized Bradley-Terry models.
The paper tackles the problem that many comparison-based preference learning models lack monotonicity guarantees, proposing a new class of Linear Generalized Bradley-Terry models with Diffusion Priors that ensures monotonicity while enabling generalization to uncompared data, with experiments showing improved accuracy, especially with limited datasets.
If you tell a learning model that you prefer an alternative $a$ over another alternative $b$, then you probably expect the model to be monotone, that is, the valuation of $a$ increases, and that of $b$ decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.