Adversarial Style Transfer for Robust Policy Optimization in Deep Reinforcement Learning
This work addresses generalization issues in deep reinforcement learning for agents deployed in unseen environments, representing an incremental improvement over existing methods.
The paper tackles the problem of overfitting to confounding features in reinforcement learning agents by proposing a max-min game theoretic objective with a generator that perturbs observations and a policy network that resists these perturbations, resulting in improved generalization and sample efficiency on Procgen and Distracting Control Suite benchmarks compared to baselines like data augmentation.
This paper proposes an algorithm that aims to improve generalization for reinforcement learning agents by removing overfitting to confounding features. Our approach consists of a max-min game theoretic objective. A generator transfers the style of observation during reinforcement learning. An additional goal of the generator is to perturb the observation, which maximizes the agent's probability of taking a different action. In contrast, a policy network updates its parameters to minimize the effect of such perturbations, thus staying robust while maximizing the expected future reward. Based on this setup, we propose a practical deep reinforcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), to find a robust policy that generalizes to unseen environments. We evaluate our approach on Procgen and Distracting Control Suite for generalization and sample efficiency. Empirically, ARPO shows improved performance compared to a few baseline algorithms, including data augmentation.