Epistemic Risk-Sensitive Reinforcement Learning
This work addresses model uncertainty in RL for applications requiring risk-sensitive decision-making, but it appears incremental as it builds on existing risk measures and algorithms.
The paper tackles the problem of handling model uncertainty in reinforcement learning by introducing a framework that incorporates risk preferences via utility functions, allowing tunable risk-averse, neutral, or risk-taking behaviors, and shows comparisons with optimal risk-neutral policies in such environments.
We develop a framework for interacting with uncertain environments in reinforcement learning (RL) by leveraging preferences in the form of utility functions. We claim that there is value in considering different risk measures during learning. In this framework, the preference for risk can be tuned by variation of the parameter $β$ and the resulting behavior can be risk-averse, risk-neutral or risk-taking depending on the parameter choice. We evaluate our framework for learning problems with model uncertainty. We measure and control for \emph{epistemic} risk using dynamic programming (DP) and policy gradient-based algorithms. The risk-averse behavior is then compared with the behavior of the optimal risk-neutral policy in environments with epistemic risk.