Robust Reinforcement Learning for Continuous Control with Model Misspecification
This work addresses robustness to environmental perturbations for continuous control RL applications, such as robotics, but is incremental as it builds upon existing MPO methods.
The authors tackled the problem of model misspecification in continuous control reinforcement learning by developing robust and soft-robust frameworks integrated into the MPO algorithm, resulting in policies that outperformed non-robust versions in nine Mujoco domains and a high-dimensional robotic hand simulation.
We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.