Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes
This addresses risk-averse sequential decision-making for systems where catastrophic events must be avoided, representing an incremental improvement by integrating risk constraints into existing planning methods.
The paper tackles the problem of maximizing expected discounted-sum payoff in Markov decision processes while ensuring the probability of catastrophic events stays below a threshold, and presents an efficient algorithm that combines UCT-like search, a learned predictor, and linear programming, achieving results on benchmarks with up to 10^6 states.
Markov decision processes (MDPs) are the defacto frame-work for sequential decision making in the presence ofstochastic uncertainty. A classical optimization criterion forMDPs is to maximize the expected discounted-sum pay-off, which ignores low probability catastrophic events withhighly negative impact on the system. On the other hand,risk-averse policies require the probability of undesirableevents to be below a given threshold, but they do not accountfor optimization of the expected payoff. We consider MDPswith discounted-sum payoff with failure states which repre-sent catastrophic outcomes. The objective of risk-constrainedplanning is to maximize the expected discounted-sum payoffamong risk-averse policies that ensure the probability to en-counter a failure state is below a desired threshold. Our maincontribution is an efficient risk-constrained planning algo-rithm that combines UCT-like search with a predictor learnedthrough interaction with the MDP (in the style of AlphaZero)and with a risk-constrained action selection via linear pro-gramming. We demonstrate the effectiveness of our approachwith experiments on classical MDPs from the literature, in-cluding benchmarks with an order of 10^6 states.