Offline-to-online hyperparameter transfer for stochastic bandits
This addresses a practical issue for practitioners using bandit algorithms, offering a method to reduce tuning effort, but it is incremental as it builds on existing bandit frameworks.
The paper tackles the problem of tuning hyperparameters for stochastic bandit algorithms, which is challenging and sometimes impossible, by proposing a transfer learning approach using offline data from multiple tasks to set hyperparameters for new tasks, with theoretical bounds on sample complexity and experimental validation.
Classic algorithms for stochastic bandits typically use hyperparameters that govern their critical properties such as the trade-off between exploration and exploitation. Tuning these hyperparameters is a problem of great practical significance. However, this is a challenging problem and in certain cases is information theoretically impossible. To address this challenge, we consider a practically relevant transfer learning setting where one has access to offline data collected from several bandit problems (tasks) coming from an unknown distribution over the tasks. Our aim is to use this offline data to set the hyperparameters for a new task drawn from the unknown distribution. We provide bounds on the inter-task (number of tasks) and intra-task (number of arm pulls for each task) sample complexity for learning near-optimal hyperparameters on unseen tasks drawn from the distribution. Our results apply to several classic algorithms, including tuning the exploration parameters in UCB and LinUCB and the noise parameter in GP-UCB. Our experiments indicate the significance and effectiveness of the transfer of hyperparameters from offline problems in online learning with stochastic bandit feedback.