Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
This work addresses the limitation of linear reward assumptions in applications like online recommendations, offering a more flexible approach for handling complex preferences.
The paper tackles the problem of preference-based optimization in contextual dueling bandits by using a neural network to estimate non-linear reward functions from human feedback, achieving sub-linear regret guarantees with proposed algorithms.
Contextual dueling bandit is used to model the bandit problems, where a learner's goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can be complex and non-linear in many real-life applications like online recommendations or ranking web search results. To overcome this challenge, we use a neural network to estimate the reward function using preference feedback for the previously selected arms. We propose upper confidence bound- and Thompson sampling-based algorithms with sub-linear regret guarantees that efficiently select arms in each round. We also extend our theoretical results to contextual bandit problems with binary feedback, which is in itself a non-trivial contribution. Experimental results on the problem instances derived from synthetic datasets corroborate our theoretical results.