Robust Reinforcement Learning with Distributional Risk-averse formulation
This work addresses robustness in reinforcement learning for environments with uncertain dynamics or rewards, but it appears incremental as it builds on existing distributional methods.
The paper tackles robust reinforcement learning by approximating a robust formulation with a risk-averse approach using standard deviation penalization, proposing two distributional algorithms for discrete and continuous action spaces, and demonstrating their robustness in Gym environments.
Robust Reinforcement Learning tries to make predictions more robust to changes in the dynamics or rewards of the system. This problem is particularly important when the dynamics and rewards of the environment are estimated from the data. In this paper, we approximate the Robust Reinforcement Learning constrained with a $Φ$-divergence using an approximate Risk-Averse formulation. We show that the classical Reinforcement Learning formulation can be robustified using standard deviation penalization of the objective. Two algorithms based on Distributional Reinforcement Learning, one for discrete and one for continuous action spaces are proposed and tested in a classical Gym environment to demonstrate the robustness of the algorithms.