RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk
This work addresses safety in reinforcement learning for applications requiring robust decision-making under combined uncertainties, representing an incremental advance by integrating existing risk-averse and soft-robust methods.
The paper tackles the problem of jointly modeling risk from aleatory and epistemic uncertainties in reinforcement learning, proposing the RASR framework, which enables efficient computation of optimal deterministic time-dependent policies using dynamic programming with EVaR or entropic risk measures, and empirical results show consistent mitigation of uncertainty as measured by standard risk metrics.
Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs. We call this framework that combines Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion is defined using either EVaR or the entropic risk, the optimal policy in RASR can be computed efficiently using a new dynamic program formulation with a time-dependent risk level. As a result, the optimal risk-averse policies are deterministic but time-dependent, even in the infinite-horizon discounted setting. We also show that particular RASR objectives reduce to risk-averse RL with mean posterior transition probabilities. Our empirical results show that our new algorithms consistently mitigate uncertainty as measured by EVaR and other standard risk measures.