Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning
This addresses the problem of scalable skill acquisition for general-purpose agents, representing an incremental improvement with specific gains in robotics simulations.
The paper tackles the challenge of learning large skill repertoires by introducing Hierarchical Empowerment, a framework that makes empowerment-based skill learning more tractable through a new variational bound and hierarchical architecture, achieving over two orders of magnitude larger skill coverage in ant navigation tasks compared to prior work.
General purpose agents will require large repertoires of skills. Empowerment -- the maximum mutual information between skills and states -- provides a pathway for learning large collections of distinct skills, but mutual information is difficult to optimize. We introduce a new framework, Hierarchical Empowerment, that makes computing empowerment more tractable by integrating concepts from Goal-Conditioned Hierarchical Reinforcement Learning. Our framework makes two specific contributions. First, we introduce a new variational lower bound on mutual information that can be used to compute empowerment over short horizons. Second, we introduce a hierarchical architecture for computing empowerment over exponentially longer time scales. We verify the contributions of the framework in a series of simulated robotics tasks. In a popular ant navigation domain, our four level agents are able to learn skills that cover a surface area over two orders of magnitude larger than prior work.