Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback
This work addresses the problem of reducing human effort in personalization systems, offering an incremental improvement over existing methods by avoiding parametric model assumptions.
The paper tackles the inefficiency of dueling bandit algorithms in interactive preference elicitation when human feedback is sparse, by introducing a model-free approach with augmented confidence bounds that achieves competitive performance across benchmarks like recommendation and LLM response optimization.
Interactive preference elicitation (IPE) aims to substantially reduce human effort while acquiring human preferences in wide personalization systems. Dueling bandit (DB) algorithms enable optimal decision-making in IPE building on pairwise comparisons. However, they remain inefficient when human feedback is sparse. Existing methods address sparsity by heavily relying on parametric reward models, whose rigid assumptions are vulnerable to misspecification. In contrast, we explore an alternative perspective based on feedback augmentation, and introduce critical improvements to the model-free DB framework. Specifically, we introduce augmented confidence bounds to integrate augmented human feedback under generalized concentration properties, and analyze the multi-factored performance trade-off via regret analysis. Our prototype algorithm achieves competitive performance across several IPE benchmarks, including recommendation, multi-objective optimization, and response optimization for large language models, demonstrating the potential of our approach for provably efficient IPE in broader applications.