Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning
This provides a theoretical foundation for representation learning in RL, addressing a gap in understanding for researchers in self-supervised RL, though it is incremental as it builds on existing MISL methods.
The paper tackles the problem of understanding how mutual information skill learning (MISL) methods in reinforcement learning can recover ground-truth environment features, proving that Contrastive Successor Features (CSF) can provably recover these features up to a linear transformation due to skill diversity and inner product parametrization, with empirical validation in MuJoCo and DeepMind Control showing recovery from states and pixels.
Self-supervised feature learning and pretraining methods in reinforcement learning (RL) often rely on information-theoretic principles, termed mutual information skill learning (MISL). These methods aim to learn a representation of the environment while also incentivizing exploration thereof. However, the role of the representation and mutual information parametrization in MISL is not yet well understood theoretically. Our work investigates MISL through the lens of identifiable representation learning by focusing on the Contrastive Successor Features (CSF) method. We prove that CSF can provably recover the environment's ground-truth features up to a linear transformation due to the inner product parametrization of the features and skill diversity in a discriminative sense. This first identifiability guarantee for representation learning in RL also helps explain the implications of different mutual information objectives and the downsides of entropy regularizers. We empirically validate our claims in MuJoCo and DeepMind Control and show how CSF provably recovers the ground-truth features both from states and pixels.