On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes
This work addresses a gap in dynamic programming algorithms for risk-sensitive MDPs, providing a convergence proof that could benefit researchers and practitioners in robust decision-making under uncertainty, though it is incremental as it extends known methods to a new problem variant.
The authors tackled the convergence of Modified Policy Iteration (MPI) in risk-sensitive exponential cost Markov Decision Processes, proving for the first time that MPI converges in finite state and action spaces and demonstrating through simulations that it offers enhanced computational efficiency compared to value and policy iteration.
Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI has been well studied in the context of discounted and average-cost MDPs. In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration have been well studied in the context of risk sensitive MDPs, MPI is unexplored. We provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Since the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof which is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk sensitive value and policy iteration approaches. We conclude our analysis with simulation results, assessing MPI's performance relative to alternative dynamic programming methods like value iteration and policy iteration across diverse problem parameters. Our findings highlight risk-sensitive MPI's enhanced computational efficiency compared to both value and policy iteration techniques.