Landmark Guided Active Exploration with State-specific Balance Coefficient
This work addresses sample efficiency issues in hierarchical reinforcement learning for long-horizon tasks, representing an incremental improvement over existing methods.
The paper tackles the challenge of inefficient exploration in goal-conditioned hierarchical reinforcement learning due to large action spaces by proposing a landmark-guided exploration strategy with a state-specific balance coefficient, resulting in significantly outperforming baseline methods across multiple tasks.
Goal-conditioned hierarchical reinforcement learning (GCHRL) decomposes long-horizon tasks into sub-tasks through a hierarchical framework and it has demonstrated promising results across a variety of domains. However, the high-level policy's action space is often excessively large, presenting a significant challenge to effective exploration and resulting in potentially inefficient training. In this paper, we design a measure of prospect for sub-goals by planning in the goal space based on the goal-conditioned value function. Building upon the measure of prospect, we propose a landmark-guided exploration strategy by integrating the measures of prospect and novelty which aims to guide the agent to explore efficiently and improve sample efficiency. In order to dynamically consider the impact of prospect and novelty on exploration, we introduce a state-specific balance coefficient to balance the significance of prospect and novelty. The experimental results demonstrate that our proposed exploration strategy significantly outperforms the baseline methods across multiple tasks.