EUBRL: Epistemic Uncertainty Directed Bayesian Reinforcement Learning
This addresses the problem of efficient exploration in reinforcement learning for agents in complex environments, representing a novel method for a known bottleneck.
The paper tackled the exploration-exploitation dilemma in reinforcement learning by proposing EUBRL, a Bayesian RL algorithm that uses epistemic uncertainty guidance to reduce per-step regret, achieving nearly minimax-optimal regret and sample complexity guarantees and demonstrating superior sample efficiency, scalability, and consistency in sparse-reward, long-horizon tasks.
At the boundary between the known and the unknown, an agent inevitably confronts the dilemma of whether to explore or to exploit. Epistemic uncertainty reflects such boundaries, representing systematic uncertainty due to limited knowledge. In this paper, we propose a Bayesian reinforcement learning (RL) algorithm, $\texttt{EUBRL}$, which leverages epistemic guidance to achieve principled exploration. This guidance adaptively reduces per-step regret arising from estimation errors. We establish nearly minimax-optimal regret and sample complexity guarantees for a class of sufficiently expressive priors in infinite-horizon discounted MDPs. Empirically, we evaluate $\texttt{EUBRL}$ on tasks characterized by sparse rewards, long horizons, and stochasticity. Results demonstrate that $\texttt{EUBRL}$ achieves superior sample efficiency, scalability, and consistency.