Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning
This work addresses a specific bottleneck in reinforcement learning for researchers and practitioners, offering an incremental improvement to enhance exploration and efficiency in tasks with sparse rewards.
The paper tackled the problem of making potential-based reward shaping more effective in reinforcement learning by showing how a linear shift of the potential function can improve sample efficiency without altering policy invariance, and demonstrated this empirically on Gridworld, Cart Pole, and Mountain Car environments.
Potential-based reward shaping is commonly used to incorporate prior knowledge of how to solve the task into reinforcement learning because it can formally guarantee policy invariance. As such, the optimal policy and the ordering of policies by their returns are not altered by potential-based reward shaping. In this work, we highlight the dependence of effective potential-based reward shaping on the initial Q-values and external rewards, which determine the agent's ability to exploit the shaping rewards to guide its exploration and achieve increased sample efficiency. We formally derive how a simple linear shift of the potential function can be used to improve the effectiveness of reward shaping without changing the encoded preferences in the potential function, and without having to adjust the initial Q-values, which can be challenging and undesirable in deep reinforcement learning. We show the theoretical limitations of continuous potential functions for correctly assigning positive and negative reward shaping values. We verify our theoretical findings empirically on Gridworld domains with sparse and uninformative reward functions, as well as on the Cart Pole and Mountain Car environments, where we demonstrate the application of our results in deep reinforcement learning.