Hypothesis Transfer in Bandits by Weighted Models
This work addresses the challenge of improving bandit performance with pre-learned models, offering a domain-specific solution that is incremental in nature.
The paper tackles the problem of accelerating exploration in contextual multi-armed bandits using hypothesis transfer learning, showing a reduction in regret compared to Linear UCB when transfer is effective while maintaining standard rates otherwise.
We consider the problem of contextual multi-armed bandits in the setting of hypothesis transfer learning. That is, we assume having access to a previously learned model on an unobserved set of contexts, and we leverage it in order to accelerate exploration on a new bandit problem. Our transfer strategy is based on a re-weighting scheme for which we show a reduction in the regret over the classic Linear UCB when transfer is desired, while recovering the classic regret rate when the two tasks are unrelated. We further extend this method to an arbitrary amount of source models, where the algorithm decides which model is preferred at each time step. Additionally we discuss an approach where a dynamic convex combination of source models is given in terms of a biased regularization term in the classic LinUCB algorithm. The algorithms and the theoretical analysis of our proposed methods substantiated by empirical evaluations on simulated and real-world data.