Dual Goal Representations
This work addresses the challenge of learning effective goal representations in reinforcement learning, offering a novel approach that is invariant to state representation and filters noise, with incremental improvements in performance.
The paper tackles the problem of goal-conditioned reinforcement learning by introducing dual goal representations, which encode states through temporal distances to all other states, and demonstrates that this method improves offline goal-reaching performance across 20 tasks in the OGBench suite.
In this work, we introduce dual goal representations for goal-conditioned reinforcement learning (GCRL). A dual goal representation characterizes a state by "the set of temporal distances from all other states"; in other words, it encodes a state through its relations to every other state, measured by temporal distance. This representation provides several appealing theoretical properties. First, it depends only on the intrinsic dynamics of the environment and is invariant to the original state representation. Second, it contains provably sufficient information to recover an optimal goal-reaching policy, while being able to filter out exogenous noise. Based on this concept, we develop a practical goal representation learning method that can be combined with any existing GCRL algorithm. Through diverse experiments on the OGBench task suite, we empirically show that dual goal representations consistently improve offline goal-reaching performance across 20 state- and pixel-based tasks.