C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
This addresses a central problem in goal-conditioned RL for domains like navigation and manipulation, offering a more efficient and effective solution for long-horizon tasks, though it is an incremental improvement over existing methods.
The paper tackles the challenge of learning to reach distant goals in goal-conditioned reinforcement learning without offline data or expert demonstrations by proposing C-Planning, an algorithm that uses search during training to generate a curriculum of intermediate states, resulting in improved sample efficiency and the ability to solve long-horizon tasks that prior methods fail on.
Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field. Learning to reach such goals is particularly hard without any offline data, expert demonstrations, and reward shaping. In this paper, we propose an algorithm to solve the distant goal-reaching task by using search at training time to automatically generate a curriculum of intermediate states. Our algorithm, Classifier-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning an optimal sequence of waypoints using graph search, while the M-step aims to learn a goal-conditioned policy to reach those waypoints. Unlike prior methods that combine goal-conditioned RL with graph search, ours performs search only during training and not testing, significantly decreasing the compute costs of deploying the learned policy. Empirically, we demonstrate that our method is more sample efficient than prior methods. Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goal-conditioned methods and methods based on graph search fail to solve.