Exploration via Elliptical Episodic Bonuses
This addresses the challenge of effective exploration in reinforcement learning for researchers and practitioners dealing with high-dimensional, noisy environments, representing a novel extension rather than an incremental improvement.
The paper tackled the problem of exploration in reinforcement learning for complex, noisy environments by introducing a new method called E3B, which extends count-based episodic bonuses to continuous spaces using a learned embedding, resulting in state-of-the-art performance on 16 MiniHack tasks and competitive results on VizDoom and Habitat environments.
In recent years, a number of reinforcement learning (RL) methods have been proposed to explore complex environments which differ across episodes. In this work, we show that the effectiveness of these methods critically relies on a count-based episodic term in their exploration bonus. As a result, despite their success in relatively simple, noise-free settings, these methods fall short in more realistic scenarios where the state space is vast and prone to noise. To address this limitation, we introduce Exploration via Elliptical Episodic Bonuses (E3B), a new method which extends count-based episodic bonuses to continuous state spaces and encourages an agent to explore states that are diverse under a learned embedding within each episode. The embedding is learned using an inverse dynamics model in order to capture controllable aspects of the environment. Our method sets a new state-of-the-art across 16 challenging tasks from the MiniHack suite, without requiring task-specific inductive biases. E3B also matches existing methods on sparse reward, pixel-based VizDoom environments, and outperforms existing methods in reward-free exploration on Habitat, demonstrating that it can scale to high-dimensional pixel-based observations and realistic environments.