Active Reinforcement Learning with Monte-Carlo Tree Search
This addresses a fundamental exploration problem in reinforcement learning for agents operating under costly reward observations, though it is incremental as it builds on existing simulation-based methods.
The paper tackles the challenge of Active Reinforcement Learning (ARL), where agents must pay a cost to observe rewards, by proposing an algorithm based on Monte-Carlo Tree Search that is asymptotically Bayes optimal and outperforms heuristic-based methods on larger MDPs.
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular environments to Bayes-Adaptive MDPs. We provide an ARL algorithm using Monte-Carlo Tree Search that is asymptotically Bayes optimal. Experimentally, this algorithm is near-optimal on small Bandit problems and MDPs. On larger MDPs it outperforms a Q-learner augmented with specialised heuristics for ARL. By analysing exploration behaviour in detail, we uncover obstacles to scaling up simulation-based algorithms for ARL.