A Bayesian Approach to Online Planning
This work addresses the challenge of enhancing planning efficiency in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing Monte Carlo tree search and neural network methods.
The paper tackles the problem of imperfect neural network approximations in online planning by developing a Bayesian approach that uses uncertainty estimates to improve tree search, demonstrating more effective search in ProcGen Maze and Leaper environments when uncertainty estimates are accurate.
The combination of Monte Carlo tree search and neural networks has revolutionized online planning. As neural network approximations are often imperfect, we ask whether uncertainty estimates about the network outputs could be used to improve planning. We develop a Bayesian planning approach that facilitates such uncertainty quantification, inspired by classical ideas from the meta-reasoning literature. We propose a Thompson sampling based algorithm for searching the tree of possible actions, for which we prove the first (to our knowledge) finite time Bayesian regret bound, and propose an efficient implementation for a restricted family of posterior distributions. In addition we propose a variant of the Bayes-UCB method applied to trees. Empirically, we demonstrate that on the ProcGen Maze and Leaper environments, when the uncertainty estimates are accurate but the neural network output is inaccurate, our Bayesian approach searches the tree much more effectively. In addition, we investigate whether popular uncertainty estimation methods are accurate enough to yield significant gains in planning. Our code is available at: https://github.com/nirgreshler/bayesian-online-planning.