Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
This addresses a key bottleneck in goal-conditioned RL for improving learning efficiency and generalization, though it appears incremental as it builds on existing goal-conditioned policy frameworks.
The paper tackles the problem of learning appropriate distance functions for goal-conditioned reinforcement learning, which is crucial for determining goal achievement, by proposing a self-supervised method that estimates distance based on the number of actions needed, and it successfully solves complex tasks in three scenarios without prior domain knowledge.
Goal-conditioned policies are used in order to break down complex reinforcement learning (RL) problems by using subgoals, which can be defined either in state space or in a latent feature space. This can increase the efficiency of learning by using a curriculum, and also enables simultaneous learning and generalization across goals. A crucial requirement of goal-conditioned policies is to be able to determine whether the goal has been achieved. Having a notion of distance to a goal is thus a crucial component of this approach. However, it is not straightforward to come up with an appropriate distance, and in some tasks, the goal space may not even be known a priori. In this work we learn a distance-to-goal estimate which is computed in terms of the number of actions that would need to be carried out in a self-supervised approach. Our method solves complex tasks without prior domain knowledge in the online setting in three different scenarios in the context of goal-conditioned policies a) the goal space is the same as the state space b) the goal space is given but an appropriate distance is unknown and c) the state space is accessible, but only a subset of the state space represents desired goals, and this subset is known a priori. We also propose a goal-generation mechanism as a secondary contribution.