DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards
This addresses the challenge of efficient exploration in RL for agents in sparse-reward environments, representing an incremental improvement over existing novelty-based methods.
The paper tackles the problem of exploration in reinforcement learning by proposing DEIR, a method that derives an intrinsic reward based on conditional mutual information to better evaluate exploratory behaviors, and shows it learns better policies faster than baselines on MiniGrid and generalizes to ProcGen.
Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward. Our source code is available at https://github.com/swan-utokyo/deir.