Policy Evaluation with Variance Related Risk Criteria in Markov Decision Processes
This addresses risk management in domains like finance and process control, but is incremental as it extends existing temporal difference methods.
The authors tackled the problem of policy evaluation in Markov Decision Processes when performance criteria include reward variance, proposing temporal difference algorithms with linear function approximation and proving their convergence. They demonstrated utility in a 4-dimensional continuous state space problem.
In this paper we extend temporal difference policy evaluation algorithms to performance criteria that include the variance of the cumulative reward. Such criteria are useful for risk management, and are important in domains such as finance and process control. We propose both TD(0) and LSTD(lambda) variants with linear function approximation, prove their convergence, and demonstrate their utility in a 4-dimensional continuous state space problem.