Model-free Policy Learning with Reward Gradients
This work addresses sample-scarce applications like robotics by providing a model-free method to enhance policy learning, though it is incremental as it builds on existing policy gradient frameworks.
The paper tackles the sample inefficiency of policy gradient methods in reinforcement learning by introducing the Reward Policy Gradient estimator, which integrates reward gradients without requiring environment dynamics, resulting in improved sample efficiency and performance on MuJoCo control tasks.
Despite the increasing popularity of policy gradient methods, they are yet to be widely utilized in sample-scarce applications, such as robotics. The sample efficiency could be improved by making best usage of available information. As a key component in reinforcement learning, the reward function is usually devised carefully to guide the agent. Hence, the reward function is usually known, allowing access to not only scalar reward signals but also reward gradients. To benefit from reward gradients, previous works require the knowledge of environment dynamics, which are hard to obtain. In this work, we develop the \textit{Reward Policy Gradient} estimator, a novel approach that integrates reward gradients without learning a model. Bypassing the model dynamics allows our estimator to achieve a better bias-variance trade-off, which results in a higher sample efficiency, as shown in the empirical analysis. Our method also boosts the performance of Proximal Policy Optimization on different MuJoCo control tasks.