Exploiting Concavity Information in Gaussian Process Contextual Bandit Optimization
This addresses efficiency challenges in sequential decision-making for domains like medicine and advertising where concave reward structures are common, though it represents an incremental improvement over existing Bayesian bandit methods.
The paper tackles the problem of improving contextual bandit optimization by incorporating known concavity information about reward functions, achieving up to 40% reduction in cumulative regret compared to standard methods in experiments on medical dosing scenarios.
The contextual bandit framework is widely used to solve sequential optimization problems where the reward of each decision depends on auxiliary context variables. In settings such as medicine, business, and engineering, the decision maker often possesses additional structural information on the generative model that can potentially be used to improve the efficiency of bandit algorithms. We consider settings in which the mean reward is known to be a concave function of the action for each fixed context. Examples include patient-specific dose-response curves in medicine and expected profit in online advertising auctions. We propose a contextual bandit algorithm that accelerates optimization by conditioning the posterior of a Bayesian Gaussian Process model on this concavity information. We design a novel shape-constrained reward function estimator using a specially chosen regression spline basis and constrained Gaussian Process posterior. Using this model, we propose a UCB algorithm and derive corresponding regret bounds. We evaluate our algorithm on numerical examples and test functions used to study optimal dosing of Anti-Clotting medication.