Risk-Averse Reinforcement Learning with Itakura-Saito Loss
This work addresses risk management in high-stakes applications like finance or robotics, but it is incremental as it adapts an existing divergence for a specific utility case.
The paper tackles the problem of risk-averse reinforcement learning by introducing the Itakura-Saito loss function, showing that it outperforms established alternatives in experiments with known analytical solutions.
Risk-averse reinforcement learning finds application in various high-stakes fields. Unlike classical reinforcement learning, which aims to maximize expected returns, risk-averse agents choose policies that minimize risk, occasionally sacrificing expected value. These preferences can be framed through utility theory. We focus on the specific case of the exponential utility function, where one can derive the Bellman equations and employ various reinforcement learning algorithms with few modifications. To address this, we introduce to the broad machine learning community a numerically stable and mathematically sound loss function based on the Itakura-Saito divergence for learning state-value and action-value functions. We evaluate the Itakura-Saito loss function against established alternatives, both theoretically and empirically. In the experimental section, we explore multiple scenarios, some with known analytical solutions, and show that the considered loss function outperforms the alternatives.