Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
This work provides a foundational framework for representation learning in goal-based RL, which is incremental as it builds on existing methods but offers a novel perspective and tools for analysis.
The paper tackles the problem of unifying goal-conditioned reinforcement learning (GCRL) and mutual information-based RL into a single framework called variational GCRL (VGCRL), interpreting variational empowerment as representation learning for goal reaching, and demonstrates this through derivations and experiments that include new variants, adaptations of techniques like hindsight experience replay, and a novel evaluation metric.
Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MI-based RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionally-aware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine the quality of discovered skills, or latent goals, through modifying latent dimensionality and applying spectral normalization; (3) adapt techniques such as hindsight experience replay (HER) from GCRL to MI-based RL; and lastly, (4) propose a novel evaluation metric, named latent goal reaching (LGR), for comparing empowerment algorithms with different choices of latent dimensionality and discriminator parameterization. Through principled mathematical derivations and careful experimental studies, our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.