Stochastic Multi-Armed Bandits with Limited Control Variates
This addresses a practical limitation in applications like wireless networks where control variates are not always available, offering an incremental improvement over prior methods.
The paper tackles the problem of stochastic multi-armed bandits with limited access to auxiliary control variates, proposing UCB-LCV, which outperforms existing algorithms by effectively combining reward and control variate estimators.
Motivated by wireless networks where interference or channel state estimates provide partial insight into throughput, we study a variant of the classical stochastic multi-armed bandit problem in which the learner has limited access to auxiliary information. Recent work has shown that such auxiliary information, when available as control variates, can be used to get tighter confidence bounds, leading to lower regret. However, existing works assume that control variates are available in every round, which may not be realistic in several real-life scenarios. To address this, we propose UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates. When there is no control variate, UCB-LCV leads to a novel algorithm that we call UCB-NORMAL, outperforming its existing algorithms for the standard MAB setting with normally distributed rewards. Finally, we discuss variants of the proposed UCB-LCV that apply to general distributions and experimentally demonstrate that UCB-LCV outperforms existing bandit algorithms.