Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation
This work addresses sample efficiency for RL practitioners, offering an incremental improvement over existing ensemble methods.
The paper tackled the problem of noisy value estimates harming sample efficiency in deep reinforcement learning by introducing a Bayesian framework that uses uncertainty estimation to weight optimization, resulting in significant improvements in sample efficiency on control tasks.
In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.