Approximate Value Iteration for Risk-aware Markov Decision Processes
It addresses the scalability problem of risk-aware MDPs for practitioners dealing with large-scale sequential decision-making under risk.
The paper develops simulation-based approximate dynamic programming algorithms for large-scale risk-aware MDPs, providing convergence analysis and sample complexity bounds.
We consider large-scale Markov decision processes (MDPs) with a risk measure of variability in cost, under the risk-aware MDPs paradigm. Previous studies showed that risk-aware MDPs, based on a minimax approach to handling risk, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this paper, we employ an approximate dynamic programming approach, and develop a family of simulation-based algorithms to approximately solve large-scale risk-aware MDPs. In parallel, we develop a unified convergence analysis technique to derive sample complexity bounds for this new family of algorithms.