Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning
This work addresses constrained reinforcement learning problems for researchers and practitioners by introducing a novel model and algorithms, though it appears incremental as it builds on existing CMDPs and SIP tools.
The authors tackled constrained reinforcement learning by proposing a generalization called semi-infinitely constrained Markov decision processes (SICMDPs) with a continuum of constraints, and they developed two algorithms (SI-CRL and SI-CPO) that demonstrated effectiveness in solving complex sequential decision-making tasks using deep reinforcement learning techniques.
We propose a novel generalization of constrained Markov decision processes (CMDPs) that we call the \emph{semi-infinitely constrained Markov decision process} (SICMDP). Particularly, we consider a continuum of constraints instead of a finite number of constraints as in the case of ordinary CMDPs. We also devise two reinforcement learning algorithms for SICMDPs that we call SI-CRL and SI-CPO. SI-CRL is a model-based reinforcement learning algorithm. Given an estimate of the transition model, we first transform the reinforcement learning problem into a linear semi-infinitely programming (LSIP) problem and then use the dual exchange method in the LSIP literature to solve it. SI-CPO is a policy optimization algorithm. Borrowing the ideas from the cooperative stochastic approximation approach, we make alternative updates to the policy parameters to maximize the reward or minimize the cost. To the best of our knowledge, we are the first to apply tools from semi-infinitely programming (SIP) to solve constrained reinforcement learning problems. We present theoretical analysis for SI-CRL and SI-CPO, identifying their iteration complexity and sample complexity. We also conduct extensive numerical examples to illustrate the SICMDP model and demonstrate that our proposed algorithms are able to solve complex sequential decision-making tasks leveraging modern deep reinforcement learning techniques.