Reward Constrained Policy Optimization
This addresses the issue of unwanted behavior in reinforcement learning for practitioners by offering a method to enforce constraints, though it appears incremental as it builds on existing constrained optimization approaches.
The authors tackled the problem of reinforcement learning agents exploiting reward signal loopholes by introducing a multi-timescale constrained policy optimization method, RCPO, which uses an alternative penalty signal to guide policies toward constraint satisfaction, and they proved its convergence and provided empirical evidence of its effectiveness.
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.