Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learning
This work addresses efficiency challenges in goal-conditioned reinforcement learning for AI/robotics applications, representing an incremental improvement with specific gains in multi-step learning.
The paper tackled the problem of off-policy biases in multi-step goal-conditioned reinforcement learning, which hinder efficiency due to sparse rewards, by categorizing biases into 'shooting' and 'shifting' and presenting solutions to mitigate drawbacks while leveraging benefits, resulting in superior learning efficiency and performance that surpasses baselines and state-of-the-art benchmarks in ten-step scenarios.
In goal-conditioned reinforcement learning (GCRL), sparse rewards present significant challenges, often obstructing efficient learning. Although multi-step GCRL can boost this efficiency, it can also lead to off-policy biases in target values. This paper dives deep into these biases, categorizing them into two distinct categories: "shooting" and "shifting". Recognizing that certain behavior policies can hasten policy refinement, we present solutions designed to capitalize on the positive aspects of these biases while minimizing their drawbacks, enabling the use of larger step sizes to speed up GCRL. An empirical study demonstrates that our approach ensures a resilient and robust improvement, even in ten-step learning scenarios, leading to superior learning efficiency and performance that generally surpass the baseline and several state-of-the-art multi-step GCRL benchmarks.