Thompson Sampling in Non-Episodic Restless Bandits
This provides a theoretical advance for sequential decision-making in dynamic environments like wireless communication, though it is incremental as it builds on existing frameworks.
The paper tackles the problem of learning in non-episodic restless bandits with time-varying reward distributions, achieving a sub-linear regret bound of O(√T log T) for a variant of Thompson sampling, which resolves an open question from prior work limited to episodic cases.
Restless bandit problems assume time-varying reward distributions of the arms, which adds flexibility to the model but makes the analysis more challenging. We study learning algorithms over the unknown reward distributions and prove a sub-linear, $O(\sqrt{T}\log T)$, regret bound for a variant of Thompson sampling. Our analysis applies in the infinite time horizon setting, resolving the open question raised by Jung and Tewari (2019) whose analysis is limited to the episodic case. We adopt their policy mapping framework, which allows our algorithm to be efficient and simultaneously keeps the regret meaningful. Our algorithm adapts the TSDE algorithm of Ouyang et al. (2017) in a non-trivial manner to account for the special structure of restless bandits. We test our algorithm on a simulated dynamic channel access problem with several policy mappings, and the empirical regrets agree with the theoretical bound regardless of the choice of the policy mapping.