Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
This addresses the problem of efficient exploration in reinforcement learning for robotics, using more accessible data without action labels, though it appears incremental as it builds on existing off-policy actor-critic frameworks.
The paper tackles long-horizon goal-reaching policy learning from non-expert, action-free observation data by proposing a subgoal guidance strategy, achieving significant performance advantages in robotic navigation and manipulation tasks.
In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.