Randomised Bayesian Least-Squares Policy Iteration
This work proposes an incremental improvement for reinforcement learning practitioners by enhancing policy iteration with Bayesian methods for better exploration in online settings.
The paper tackles the problem of online reinforcement learning by introducing randomised Bayesian least-squares policy iteration (RBLSPI), which uses Bayesian uncertainty quantification and Thompson sampling to address the exploration-exploitation dilemma, and demonstrates its effectiveness experimentally in several environments.
We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves its policy based on an incomplete policy evaluation step. In online setting, the exploration-exploitation dilemma should be addressed as we try to discover the optimal policy by using samples collected by ourselves. RBLSPI exploits the advantage of BLSTD to quantify our uncertainty about the value function. Inspired by Thompson sampling, RBLSPI first samples a value function from a posterior distribution over value functions, and then selects actions based on the sampled value function. The effectiveness and the exploration abilities of RBLSPI are demonstrated experimentally in several environments.