Latent-Predictive Empowerment: Measuring Empowerment without a Simulator
This work addresses a scalability bottleneck in empowerment-based skill learning for AI agents, offering a more practical solution for realistic environments.
The paper tackles the challenge of scaling empowerment for learning diverse skillsets in agents by introducing Latent-Predictive Empowerment (LPE), which replaces the need for a full simulator with a simpler latent-predictive model, achieving similar skillset sizes as leading methods and outperforming other model-based approaches in high-dimensional and stochastic settings.
Empowerment has the potential to help agents learn large skillsets, but is not yet a scalable solution for training general-purpose agents. Recent empowerment methods learn diverse skillsets by maximizing the mutual information between skills and states; however, these approaches require a model of the transition dynamics, which can be challenging to learn in realistic settings with high-dimensional and stochastic observations. We present Latent-Predictive Empowerment (LPE), an algorithm that can compute empowerment in a more practical manner. LPE learns large skillsets by maximizing an objective that is a principled replacement for the mutual information between skills and states and that only requires a simpler latent-predictive model rather than a full simulator of the environment. We show empirically in a variety of settings--including ones with high-dimensional observations and highly stochastic transition dynamics--that our empowerment objective (i) learns similar-sized skillsets as the leading empowerment algorithm that assumes access to a model of the transition dynamics and (ii) outperforms other model-based approaches to empowerment.