Algorithms for CVaR Optimization in MDPs
This work addresses risk management in MDPs for applications in finance and operations research, but it is incremental as it builds on existing CVaR methods by adapting them to policy optimization.
The paper tackles the problem of optimizing conditional value-at-risk (CVaR) in Markov decision processes (MDPs) to manage risk in sequential decision-making, resulting in the development of policy gradient and actor-critic algorithms that converge to locally optimal policies and are demonstrated in an optimal stopping problem.
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.