Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL
This work addresses goal-conditioned reinforcement learning, particularly in offline settings, offering an incremental improvement over prior methods for researchers and practitioners in robotics and AI.
The paper tackles the problem of solving goal-conditioned tasks with sparse rewards by extending Goal-Conditioned Supervised Learning (GCSL) into Weighted GCSL (WGCSL), a novel offline reinforcement learning algorithm, and demonstrates that WGCSL consistently outperforms GCSL and existing state-of-the-art offline methods in benchmark experiments.
Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL -- optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.