Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
This work addresses the problem of robustness in RL for applications requiring generalization to environmental shifts, though it appears incremental as it builds on existing two-player policy gradient methods.
The paper tackles the challenge of training robust reinforcement learning agents by introducing a sampling perspective using Stochastic Gradient Langevin Dynamics, resulting in a novel two-player RL algorithm that consistently outperforms existing baselines in generalization across different training and testing conditions on MuJoCo environments.
We introduce a sampling perspective to tackle the challenging task of training robust Reinforcement Learning (RL) agents. Leveraging the powerful Stochastic Gradient Langevin Dynamics, we present a novel, scalable two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our algorithm consistently outperforms existing baselines, in terms of generalization across different training and testing conditions, on several MuJoCo environments. Our experiments also show that, even for objective functions that entirely ignore potential environmental shifts, our sampling approach remains highly robust in comparison to standard RL algorithms.