Local Feature Swapping for Generalization in Reinforcement Learning
This addresses generalization issues in reinforcement learning for visual tasks, but it is incremental as it builds on existing regularization approaches.
The paper tackled the problem of overfitting and lack of generalization in reinforcement learning agents using visual inputs by introducing a regularization technique called CLOP, which improved robustness to visual changes and outperformed other state-of-the-art methods on the OpenAI Procgen Benchmark.
Over the past few years, the acceleration of computing resources and research in deep learning has led to significant practical successes in a range of tasks, including in particular in computer vision. Building on these advances, reinforcement learning has also seen a leap forward with the emergence of agents capable of making decisions directly from visual observations. Despite these successes, the over-parametrization of neural architectures leads to memorization of the data used during training and thus to a lack of generalization. Reinforcement learning agents based on visual inputs also suffer from this phenomenon by erroneously correlating rewards with unrelated visual features such as background elements. To alleviate this problem, we introduce a new regularization technique consisting of channel-consistent local permutations (CLOP) of the feature maps. The proposed permutations induce robustness to spatial correlations and help prevent overfitting behaviors in RL. We demonstrate, on the OpenAI Procgen Benchmark, that RL agents trained with the CLOP method exhibit robustness to visual changes and better generalization properties than agents trained using other state-of-the-art regularization techniques. We also demonstrate the effectiveness of CLOP as a general regularization technique in supervised learning.