Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
This work provides incremental insights into the mechanisms of exploration in goal-conditioned RL, benefiting researchers in reinforcement learning by clarifying how self-supervised algorithms achieve exploration without external rewards.
The paper tackled the problem of understanding emergent exploration in unsupervised reinforcement learning by analyzing the Single-Goal Contrastive Reinforcement Learning (SGCRL) algorithm, showing that it maximizes implicit rewards from learned representations to promote exploration and exploitation, with experiments revealing that exploration arises from low-rank state representations rather than neural network approximation.
In this work, we take a first step toward elucidating the mechanisms behind emergent exploration in unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL), a self-supervised algorithm capable of solving challenging long-horizon goal-reaching tasks without external rewards or curricula. We combine theoretical analysis of the algorithm's objective function with controlled experiments to understand what drives its exploration. We show that SGCRL maximizes implicit rewards shaped by its learned representations. These representations automatically modify the reward landscape to promote exploration before reaching the goal and exploitation thereafter. Our experiments also demonstrate that these exploration dynamics arise from learning low-rank representations of the state space rather than from neural network function approximation. Our improved understanding enables us to adapt SGCRL to perform safety-aware exploration.