On Efficient Bayesian Exploration in Model-Based Reinforcement Learning
This work addresses the problem of sample efficiency in reinforcement learning for researchers and practitioners, offering theoretical grounding and practical improvements, though it is incremental as it builds on existing information-theoretic methods.
The paper tackles data-efficient exploration in reinforcement learning by analyzing information-theoretic bonuses that target epistemic uncertainty, proving they converge to zero with certainty and providing formal guarantees for previously ungrounded approaches. It introduces PTS-BE, a framework integrating model-based planning with these bonuses, which empirically outperforms baselines in sparse-reward and exploratory environments.
In this work, we address the challenge of data-efficient exploration in reinforcement learning by examining existing principled, information-theoretic approaches to intrinsic motivation. Specifically, we focus on a class of exploration bonuses that targets epistemic uncertainty rather than the aleatoric noise inherent in the environment. We prove that these bonuses naturally signal epistemic information gains and converge to zero once the agent becomes sufficiently certain about the environment's dynamics and rewards, thereby aligning exploration with genuine knowledge gaps. Our analysis provides formal guarantees for IG-based approaches, which previously lacked theoretical grounding. To enable practical use, we also discuss tractable approximations via sparse variational Gaussian Processes, Deep Kernels and Deep Ensemble models. We then outline a general framework - Predictive Trajectory Sampling with Bayesian Exploration (PTS-BE) - which integrates model-based planning with information-theoretic bonuses to achieve sample-efficient deep exploration. We empirically demonstrate that PTS-BE substantially outperforms other baselines across a variety of environments characterized by sparse rewards and/or purely exploratory tasks.