Self-Imitation Advantage Learning
This work provides an incremental improvement for researchers and practitioners working with off-policy reinforcement learning, particularly in environments with hard exploration and sparse rewards.
This paper introduces Self-Imitation Advantage Learning (SAIL), a novel method that extends self-imitation learning to off-policy reinforcement learning. SAIL addresses the issue of stale returns by using the more optimistic estimate between the observed return and the current action-value, demonstrating empirical effectiveness on hard exploration games in the Arcade Learning Environment.
Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.