Estimating Risk and Uncertainty in Deep Reinforcement Learning
This work addresses risk-sensitive applications in reinforcement learning by providing a method to handle uncertainty, though it appears incremental as it builds on existing DQN variants.
The paper tackles the problem of estimating both epistemic and aleatoric uncertainty in deep reinforcement learning, proposing a framework that disentangles these uncertainties on Q-values and demonstrates improved performance with an uncertainty-aware DQN algorithm on the MinAtar testbed.
Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultaneously estimating both of them, and propose a framework for disentangling and estimating these uncertainties on learned Q-values. We derive unbiased estimators of these uncertainties and introduce an uncertainty-aware DQN algorithm, which we show exhibits safe learning behavior and outperforms other DQN variants on the MinAtar testbed.