Deep Hierarchical Planning from Pixels
This addresses the challenge of scaling AI to complex, long-sequence tasks without manual specification, which is crucial for robotics and general AI applications, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of enabling intelligent agents to perform long-horizon tasks by introducing Director, a hierarchical reinforcement learning method that learns from pixels using a world model, achieving superior performance in sparse-reward environments like 3D maze traversal with a robot and across various domains such as Atari games.
Intelligent agents need to select long sequences of actions to solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view that was used by prior work. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.