Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis
This work addresses the challenge of manual goal representation in HRL for open-ended learning, offering an incremental improvement by automating subgoal discovery to enhance efficiency and transferability.
The paper tackles the problem of autonomously discovering symbolic goal representations in hierarchical reinforcement learning, which traditionally require manual design, by proposing a developmental mechanism for subgoal discovery that groups states with similar roles, resulting in an interpretable representation and improved data efficiency in navigation tasks.
Open-ended learning benefits immensely from the use of symbolic methods for goal representation as they offer ways to structure knowledge for efficient and transferable learning. However, the existing Hierarchical Reinforcement Learning (HRL) approaches relying on symbolic reasoning are often limited as they require a manual goal representation. The challenge in autonomously discovering a symbolic goal representation is that it must preserve critical information, such as the environment dynamics. In this work, we propose a developmental mechanism for subgoal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states that have similar roles in the task. We create a HRL algorithm that gradually learns this representation along with the policies and evaluate it on navigation tasks to show the learned representation is interpretable and results in data efficiency.