Learning Compositional Neural Programs for Continuous Control
This addresses hierarchical planning in robotics for sparse-reward tasks, representing an incremental improvement by extending AlphaNPI with self-models for imagination-based planning.
The paper tackles challenging sparse-reward, continuous control problems like stacking multiple blocks by proposing AlphaNPI-X, a method that learns atomic policies, self-models, and recursive compositional programs, enabling effective task completion where model-free baselines fail.
We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple levels of abstraction. The key insight is that the self-models enable planning by imagination, obviating the need for interaction with the world when learning higher-level compositional programs. To accomplish the third stage of learning, we extend the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural programmer-interpreters. We empirically show that AlphaNPI-X can effectively learn to tackle challenging sparse manipulation tasks, such as stacking multiple blocks, where powerful model-free baselines fail.