The Role of Diverse Replay for Generalisation in Reinforcement Learning
This work addresses the challenge of generalization in reinforcement learning for multi-task settings, offering an incremental improvement over existing methods by focusing on data diversity.
The paper tackles the problem of improving zero-shot generalization in multi-task reinforcement learning by increasing the diversity of data in the replay buffer, showing empirically that this strategy enhances performance on both reachable and unreachable tasks, with concrete improvements observed in generalization metrics.
In reinforcement learning (RL), key components of many algorithms are the exploration strategy and replay buffer. These strategies regulate what environment data is collected and trained on and have been extensively studied in the RL literature. In this paper, we investigate the impact of these components in the context of generalisation in multi-task RL. We investigate the hypothesis that collecting and training on more diverse data from the training environments will improve zero-shot generalisation to new tasks. We motivate mathematically and show empirically that generalisation to tasks that are "reachable'' during training is improved by increasing the diversity of transitions in the replay buffer. Furthermore, we show empirically that this same strategy also shows improvement for generalisation to similar but "unreachable'' tasks which could be due to improved generalisation of the learned latent representations.