Soft-Robust Actor-Critic Policy-Gradient
This addresses the issue of conservativeness in robust reinforcement learning for applications requiring model uncertainty handling, but it appears incremental as it builds on existing robust formulations.
The paper tackles the problem of overly conservative robust policies in reinforcement learning by introducing a soft-robust framework that learns optimal policies with respect to a distribution over uncertainty sets, avoiding conservativeness while maintaining robustness to model uncertainty.
Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this paper, we present a novel Soft-Robust Actor-Critic algorithm (SR-AC). It learns an optimal policy with respect to a distribution over an uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust strategies. We show the convergence of SR-AC and test the efficiency of our approach on different domains by comparing it against regular learning methods and their robust formulations.