Diverse Priors for Deep Reinforcement Learning
This work addresses the exploration-exploitation dilemma in reinforcement learning, offering an incremental improvement over existing ensemble-based methods for researchers and practitioners in the field.
The paper tackles the problem of improving exploration in deep reinforcement learning by introducing a method that uses carefully designed prior neural networks to maximize diversity in initial value functions, resulting in superior performance and significantly improved sample efficiency compared to random prior approaches.
In Reinforcement Learning (RL), agents aim at maximizing cumulative rewards in a given environment. During the learning process, RL agents face the dilemma of exploitation and exploration: leveraging existing knowledge to acquire rewards or seeking potentially higher ones. Using uncertainty as a guiding principle provides an active and effective approach to solving this dilemma and ensemble-based methods are one of the prominent avenues for quantifying uncertainty. Nevertheless, conventional ensemble-based uncertainty estimation lacks an explicit prior, deviating from Bayesian principles. Besides, this method requires diversity among members to generate less biased uncertainty estimation results. To address the above problems, previous research has incorporated random functions as priors. Building upon these foundational efforts, our work introduces an innovative approach with delicately designed prior NNs, which can incorporate maximal diversity in the initial value functions of RL. Our method has demonstrated superior performance compared with the random prior approaches in solving classic control problems and general exploration tasks, significantly improving sample efficiency.