Bayesian Curiosity for Efficient Exploration in Reinforcement Learning
This work addresses the challenge of efficient exploration for reinforcement learning practitioners, offering a computationally efficient and broadly applicable solution, though it is incremental as it builds on existing algorithms.
The paper tackled the problem of high sample complexity in reinforcement learning by introducing a Bayesian curiosity method that encourages exploration of unexplored state spaces, resulting in significant improvements in sample complexity across various algorithms and tasks.
Balancing exploration and exploitation is a fundamental part of reinforcement learning, yet most state-of-the-art algorithms use a naive exploration protocol like $ε$-greedy. This contributes to the problem of high sample complexity, as the algorithm wastes effort by repeatedly visiting parts of the state space that have already been explored. We introduce a novel method based on Bayesian linear regression and latent space embedding to generate an intrinsic reward signal that encourages the learning agent to seek out unexplored parts of the state space. This method is computationally efficient, simple to implement, and can extend any state-of-the-art reinforcement learning algorithm. We evaluate the method on a range of algorithms and challenging control tasks, on both simulated and physical robots, demonstrating how the proposed method can significantly improve sample complexity.