Dynamic Bottleneck for Robust Self-Supervised Exploration
This work addresses robustness in exploration for reinforcement learning agents, particularly in noisy settings, representing an incremental improvement over existing methods.
The paper tackles the problem of exploration in reinforcement learning with sparse rewards being sensitive to dynamics-irrelevant information like noise, and proposes a Dynamic Bottleneck model to address this, resulting in outperformance over state-of-the-art methods in noisy Atari environments.
Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics-irrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.