Reward Prediction Error as an Exploration Objective in Deep RL
This addresses the problem of inefficient exploration in deep RL for tasks like goal-conditioned environments or escaping local maxima, offering a novel approach that could enhance performance in specific domains, though it appears incremental as it builds on existing exploration methods.
The paper tackles the challenge of exploration in reinforcement learning by proposing a new objective that maximizes reward prediction error (RPE) to improve efficiency in tasks where state novelty does not correlate well with reward, and demonstrates that their method QXplore outperforms baseline state-novelty methods in such scenarios.
A major challenge in reinforcement learning is exploration, when local dithering methods such as epsilon-greedy sampling are insufficient to solve a given task. Many recent methods have proposed to intrinsically motivate an agent to seek novel states, driving the agent to discover improved reward. However, while state-novelty exploration methods are suitable for tasks where novel observations correlate well with improved reward, they may not explore more efficiently than epsilon-greedy approaches in environments where the two are not well-correlated. In this paper, we distinguish between exploration tasks in which seeking novel states aids in finding new reward, and those where it does not, such as goal-conditioned tasks and escaping local reward maxima. We propose a new exploration objective, maximizing the reward prediction error (RPE) of a value function trained to predict extrinsic reward. We then propose a deep reinforcement learning method, QXplore, which exploits the temporal difference error of a Q-function to solve hard exploration tasks in high-dimensional MDPs. We demonstrate the exploration behavior of QXplore on several OpenAI Gym MuJoCo tasks and Atari games and observe that QXplore is comparable to or better than a baseline state-novelty method in all cases, outperforming the baseline on tasks where state novelty is not well-correlated with improved reward.