On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning
This work addresses sample inefficiency in reinforcement learning, offering incremental improvements for researchers and practitioners by refining PBRS techniques.
The paper tackles the challenge of selecting potential functions in Potential-Based Reward Shaping (PBRS) for reinforcement learning, analyzing bias from finite horizons and using abstractions to approximate optimal value functions; experimental results show that a simple fully-connected network can achieve performance comparable to CNN-based solutions on four environments.
The use of Potential-Based Reward Shaping (PBRS) has shown great promise in the ongoing research effort to tackle sample inefficiency in Reinforcement Learning (RL). However, choosing the right potential function remains an open challenge. Additionally, RL techniques are usually constrained to use a finite horizon for computational limitations, which introduces a bias when using PBRS. In this paper, we first build some theoretically-grounded intuition on why selecting the potential function as the optimal value function of the task at hand produces performance advantages. We then analyse the bias induced by finite horizons in the context of PBRS producing novel insights. Finally, leveraging abstractions as a way to approximate the optimal value function of the given task, we assess the sample efficiency and performance impact of PBRS on four environments including a goal-oriented navigation task and three Arcade Learning Environments (ALE) games. Remarkably, experimental results show that we can reach the same level of performance as CNN-based solutions with a simple fully-connected network.