Variance-Based Risk Estimations in Markov Processes via Transformation with State Lumping
This work addresses risk-sensitive reinforcement learning for applications requiring robust decision-making, but it is incremental as it builds on existing transformation methods.
The paper tackles the problem of estimating variance-based risk measures in Markov decision processes by using a state-augmentation transformation and proposes state lumping to reduce computational complexity, demonstrating its validity in numerical experiments.
Variance plays a crucial role in risk-sensitive reinforcement learning, and most risk measures can be analyzed via variance. In this paper, we consider two law-invariant risks as examples: mean-variance risk and exponential utility risk. With the aid of the state-augmentation transformation (SAT), we show that, the two risks can be estimated in Markov decision processes (MDPs) with a stochastic transition-based reward and a randomized policy. To relieve the enlarged state space, a novel definition of isotopic states is proposed for state lumping, considering the special structure of the transformed transition probability. In the numerical experiment, we illustrate state lumping in the SAT, errors from a naive reward simplification, and the validity of the SAT for the two risk estimations.