Risk-Averse Trust Region Optimization for Reward-Volatility Reduction
This work addresses risk aversion for decision-making in domains like finance and robotics, but it is incremental as it adapts existing methods like TRPO to a new risk measure.
The paper tackles the problem of controlling risk in reinforcement learning by introducing a novel measure called reward volatility, which bounds return variance, and develops a risk-averse actor-only algorithm based on policy gradients with monotonic improvement guarantees, achieving competitive performance in simulated financial environments.
In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. We show that the reward volatility bounds the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship, and develop an actor-only algorithm. Furthermore, thanks to the linearity of the Bellman equations defined under the new objective function, it is possible to adapt the well-known policy gradient algorithms with monotonic improvement guarantees such as TRPO in a risk-averse manner. Finally, we test the proposed approach in two simulated financial environments.