Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints
This work addresses a key bottleneck in CRL for safe and efficient agent learning, though it appears incremental as an improvement over existing penalty-based methods.
The paper tackles the challenge of selecting appropriate penalty functions in Constrained Reinforcement Learning (CRL) to balance policy performance and constraint satisfaction efficiently, proposing Exterior Penalty Policy Optimization (EPO) with a Penalty Metric Network (PMN) that adapts to constraint violations, resulting in outperforming baselines in policy performance and constraint satisfaction with stable training on complex tasks.
In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. The penalty function method has recently been studied as an effective approach for handling constraints, which imposes constraints penalties on the objective to transform the constrained problem into an unconstrained one. However, it is challenging to choose appropriate penalties that balance policy performance and constraint satisfaction efficiently. In this paper, we propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN). PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration. We theoretically prove that EPO consistently improves constraint satisfaction with a convergence guarantee. We propose a new surrogate function and provide worst-case constraint violation and approximation error. In practice, we propose an effective smooth penalty function, which can be easily implemented with a first-order optimizer. Extensive experiments are conducted, showing that EPO outperforms the baselines in terms of policy performance and constraint satisfaction with a stable training process, particularly on complex tasks.