Constrained Reinforcement Learning Under Model Mismatch
This addresses the challenge of ensuring constraint satisfaction in real-world RL deployments where model uncertainty exists, representing an incremental advance in robust constrained RL methods.
The paper tackles the problem of constrained reinforcement learning under model mismatch, where policies trained in one environment may violate constraints when deployed in a different real environment, and proposes the Robust Constrained Policy Optimization (RCPO) algorithm, which is the first to handle large/continuous state spaces with theoretical guarantees on worst-case reward improvement and constraint violation during training.
Existing studies on constrained reinforcement learning (RL) may obtain a well-performing policy in the training environment. However, when deployed in a real environment, it may easily violate constraints that were originally satisfied during training because there might be model mismatch between the training and real environments. To address the above challenge, we formulate the problem as constrained RL under model uncertainty, where the goal is to learn a good policy that optimizes the reward and at the same time satisfy the constraint under model mismatch. We develop a Robust Constrained Policy Optimization (RCPO) algorithm, which is the first algorithm that applies to large/continuous state space and has theoretical guarantees on worst-case reward improvement and constraint violation at each iteration during the training. We demonstrate the effectiveness of our algorithm on a set of RL tasks with constraints.