Penalized Proximal Policy Optimization for Safe Reinforcement Learning
This addresses the challenge of safe reinforcement learning for real-world applications, representing an incremental improvement over existing methods.
The paper tackles the problem of safe reinforcement learning by proposing Penalized Proximal Policy Optimization (P3O), which efficiently learns optimal policies while satisfying safety constraints, and experiments show it outperforms state-of-the-art algorithms in reward improvement and constraint satisfaction on constrained locomotive tasks.
Safe reinforcement learning aims to learn the optimal policy while satisfying safety constraints, which is essential in real-world applications. However, current algorithms still struggle for efficient policy updates with hard constraint satisfaction. In this paper, we propose Penalized Proximal Policy Optimization (P3O), which solves the cumbersome constrained policy iteration via a single minimization of an equivalent unconstrained problem. Specifically, P3O utilizes a simple-yet-effective penalty function to eliminate cost constraints and removes the trust-region constraint by the clipped surrogate objective. We theoretically prove the exactness of the proposed method with a finite penalty factor and provide a worst-case analysis for approximate error when evaluated on sample trajectories. Moreover, we extend P3O to more challenging multi-constraint and multi-agent scenarios which are less studied in previous work. Extensive experiments show that P3O outperforms state-of-the-art algorithms with respect to both reward improvement and constraint satisfaction on a set of constrained locomotive tasks.