Solution Methods for Constrained Markov Decision Process with Continuous Probability Modulation
This addresses a previously-unsolved problem in constrained MDPs with large action sets, offering practical methods for applications like financial portfolio management.
The paper tackles constrained Markov Decision Processes with continuous probability modulation by showing that continuous action sets can be replaced by extreme points for linear rewards and developing tractable formulations for concave and non-concave rewards, evaluating it on loan delinquency management.
We propose solution methods for previously-unsolved constrained MDPs in which actions can continuously modify the transition probabilities within some acceptable sets. While many methods have been proposed to solve regular MDPs with large state sets, there are few practical approaches for solving constrained MDPs with large action sets. In particular, we show that the continuous action sets can be replaced by their extreme points when the rewards are linear in the modulation. We also develop a tractable optimization formulation for concave reward functions and, surprisingly, also extend it to non- concave reward functions by using their concave envelopes. We evaluate the effectiveness of the approach on the problem of managing delinquencies in a portfolio of loans.