Central Path Proximal Policy Optimization
This addresses the challenge of constraint enforcement in reinforcement learning for practitioners, though it appears incremental as it builds on prior work on policy geometry and barrier methods.
The paper tackled the problem of enforcing constraints in constrained Markov decision processes without compromising final return by introducing Central Path Proximal Policy Optimization (C3PO), a modification of PPO that keeps policy iterates close to the central path, resulting in improved performance with tighter constraint enforcement compared to existing on-policy methods.
In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly into the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of the PPO loss that produces policy iterates, that stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.