Bootstrapping Upper Confidence Bound
This addresses the problem of inefficient exploration in bandit algorithms for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the over-exploration problem in Upper Confidence Bound (UCB) methods for online decision-making by proposing a non-parametric, data-dependent UCB algorithm using multiplier bootstrap with second-order correction, resulting in significant regret reductions in multi-armed and linear bandit problems.
Upper Confidence Bound (UCB) method is arguably the most celebrated one used in online decision making with partial information feedback. Existing techniques for constructing confidence bounds are typically built upon various concentration inequalities, which thus lead to over-exploration. In this paper, we propose a non-parametric and data-dependent UCB algorithm based on the multiplier bootstrap. To improve its finite sample performance, we further incorporate second-order correction into the above construction. In theory, we derive both problem-dependent and problem-independent regret bounds for multi-armed bandits under a much weaker tail assumption than the standard sub-Gaussianity. Numerical results demonstrate significant regret reductions by our method, in comparison with several baselines in a range of multi-armed and linear bandit problems.