Knowledge Gradient for Preference Learning
This addresses a computational bottleneck in preference learning for applications like user studies or A/B testing, though it is incremental as it builds on existing Bayesian optimization frameworks.
The paper tackled the challenge of extending the knowledge gradient acquisition function to preferential Bayesian optimization, where only pairwise comparisons are available, by deriving an exact analytical solution that performs strongly on benchmark problems, often outperforming existing methods.
The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.