Model-Based Actor-Critic with Chance Constraint for Stochastic System
This work provides a more efficient and less conservative approach to ensure safety in real-world reinforcement learning applications, particularly for engineers developing autonomous systems.
This paper addresses the challenge of safety in reinforcement learning for stochastic systems by proposing a model-based chance-constrained actor-critic (CCAC) algorithm. CCAC directly optimizes the objective function and safe probability with adaptive weights, achieving a five times faster convergence rate and 100 times higher online computation efficiency compared to previous methods in a stochastic car-following task, while guaranteeing safety and improving performance.
Safety is essential for reinforcement learning (RL) applied in real-world situations. Chance constraints are suitable to represent the safety requirements in stochastic systems. Previous chance-constrained RL methods usually have a low convergence rate, or only learn a conservative policy. In this paper, we propose a model-based chance constrained actor-critic (CCAC) algorithm which can efficiently learn a safe and non-conservative policy. Different from existing methods that optimize a conservative lower bound, CCAC directly solves the original chance constrained problems, where the objective function and safe probability is simultaneously optimized with adaptive weights. In order to improve the convergence rate, CCAC utilizes the gradient of dynamic model to accelerate policy optimization. The effectiveness of CCAC is demonstrated by a stochastic car-following task. Experiments indicate that compared with previous RL methods, CCAC improves the performance while guaranteeing safety, with a five times faster convergence rate. It also has 100 times higher online computation efficiency than traditional safety techniques such as stochastic model predictive control.