Improving Robustness via Risk Averse Distributional Reinforcement Learning
This addresses robustness issues for real-world RL applications, but appears incremental as it builds on existing distributional RL frameworks.
The paper tackled the problem of reinforcement learning policies lacking robustness to uncertainties and disturbances when trained in simulations, proposing a risk-aware algorithm based on distributional RL with CVaR risk measure to learn robust policies, and validated it on multiple environments with unspecified results.
One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.