A Martingale Approach and Time-Consistent Sampling-based Algorithms for Risk Management in Stochastic Optimal Control
This provides a method for risk management in stochastic control, applicable to domains like robotics and autonomous systems, but it is incremental as it builds on existing iMDP frameworks.
The paper tackles stochastic optimal control problems with risk constraints by diffusing risk into a martingale to achieve time-consistent policies, transforming the problem into a stochastic target form. It extends the iMDP algorithm for sampling-based approximation, demonstrating probabilistic soundness and asymptotic optimality in motion planning with bounded collision probabilities.
In this paper, we consider a class of stochastic optimal control problems with risk constraints that are expressed as bounded probabilities of failure for particular initial states. We present here a martingale approach that diffuses a risk constraint into a martingale to construct time-consistent control policies. The martingale stands for the level of risk tolerance over time. By augmenting the system dynamics with the controlled martingale, the original risk-constrained problem is transformed into a stochastic target problem. We extend the incremental Markov Decision Process (iMDP) algorithm to approximate arbitrarily well an optimal feedback policy of the original problem by sampling in the augmented state space and computing proper boundary conditions for the reformulated problem. We show that the algorithm is both probabilistically sound and asymptotically optimal. The performance of the proposed algorithm is demonstrated on motion planning and control problems subject to bounded probability of collision in uncertain cluttered environments.