Rewarding Episodic Visitation Discrepancy for Exploration in Reinforcement Learning
This work addresses exploration challenges in reinforcement learning for domains like gaming and robotics, offering an incremental improvement over existing intrinsic reward methods by focusing on computational efficiency and stability.
The paper tackles the problem of exploration in deep reinforcement learning for complex environments with high-dimensional observations and sparse rewards by proposing Rewarding Episodic Visitation Discrepancy (REVD), a computation-efficient method that uses Rényi divergence-based visitation discrepancy between episodes to provide intrinsic rewards, resulting in significantly improved sample efficiency and outperforming benchmarking methods on Atari games and PyBullet Robotics Environments.
Exploration is critical for deep reinforcement learning in complex environments with high-dimensional observations and sparse rewards. To address this problem, recent approaches proposed to leverage intrinsic rewards to improve exploration, such as novelty-based exploration and prediction-based exploration. However, many intrinsic reward modules require sophisticated structures and representation learning, resulting in prohibitive computational complexity and unstable performance. In this paper, we propose Rewarding Episodic Visitation Discrepancy (REVD), a computation-efficient and quantified exploration method. More specifically, REVD provides intrinsic rewards by evaluating the Rényi divergence-based visitation discrepancy between episodes. To make efficient divergence estimation, a k-nearest neighbor estimator is utilized with a randomly-initialized state encoder. Finally, the REVD is tested on Atari games and PyBullet Robotics Environments. Extensive experiments demonstrate that REVD can significantly improves the sample efficiency of reinforcement learning algorithms and outperforms the benchmarking methods.