MDP modeling for multi-stage stochastic programs
This work addresses modeling challenges in stochastic optimization for researchers and practitioners, but it appears incremental as it builds on existing MDP and policy graph frameworks.
The paper tackles the problem of modeling multi-stage stochastic programs with features from Markov decision processes, including continuous spaces and decision-dependent uncertainty, by extending policy graphs and developing new stochastic dual dynamic programming variants to handle non-convexities.
We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous state and action spaces. We extend policy graphs to include decision-dependent uncertainty for one-step transition probabilities as well as a limited form of statistical learning. We focus on the expressiveness of our modeling approach, illustrating ideas with a series of examples of increasing complexity. As a solution method, we develop new variants of stochastic dual dynamic programming, including approximations to handle non-convexities.