Search-Based Adversarial Estimates for Improving Sample Efficiency in Off-Policy Reinforcement Learning
This addresses sample inefficiency for reinforcement learning practitioners, but it is incremental as it builds on existing feedback-based methods.
The paper tackles sample inefficiency in deep reinforcement learning, especially in sparse-reward environments, by introducing Adversarial Estimates that use latent similarity search from human-collected trajectories to boost learning, resulting in faster convergence for feedback-based algorithms.
Sample inefficiency is a long-lasting challenge in deep reinforcement learning (DRL). Despite dramatic improvements have been made, the problem is far from being solved and is especially challenging in environments with sparse or delayed rewards. In our work, we propose to use Adversarial Estimates as a new, simple and efficient approach to mitigate this problem for a class of feedback-based DRL algorithms. Our approach leverages latent similarity search from a small set of human-collected trajectories to boost learning, using only five minutes of human-recorded experience. The results of our study show algorithms trained with Adversarial Estimates converge faster than their original version. Moreover, we discuss how our approach could enable learning in feedback-based algorithms in extreme scenarios with very sparse rewards.