Verification of Markov Decision Processes with Risk-Sensitive Measures
This work addresses policy computation for risk-sensitive decision-making in MDPs, which is incremental as it builds on existing risk measures from cumulative prospect theory.
The authors tackled the problem of computing policies in Markov decision processes with risk-sensitive measures under temporal logic constraints by approximating a nonlinear weighting function as the difference of two convex functions, enabling efficient policy computation via convex-concave programming and demonstrating effectiveness in several scenarios.
We develop a method for computing policies in Markov decision processes with risk-sensitive measures subject to temporal logic constraints. Specifically, we use a particular risk-sensitive measure from cumulative prospect theory, which has been previously adopted in psychology and economics. The nonlinear transformation of the probabilities and utility functions yields a nonlinear programming problem, which makes computation of optimal policies typically challenging. We show that this nonlinear weighting function can be accurately approximated by the difference of two convex functions. This observation enables efficient policy computation using convex-concave programming. We demonstrate the effectiveness of the approach on several scenarios.