Visual Robot Task Planning
This addresses the challenge of visual task planning in robotics, which is incremental as it builds on existing imitation learning and generative modeling approaches.
The paper tackles the problem of enabling robots to plan complex tasks from visual information by learning a generative model for predicting future states after high-level actions, and demonstrates its application in navigation, pick-and-place, and surgical robotics tasks.
Prospection, the act of predicting the consequences of many possible futures, is intrinsic to human planning and action, and may even be at the root of consciousness. Surprisingly, this idea has been explored comparatively little in robotics. In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment. Models are trained via imitation learning on a variety of domains, including navigation, pick-and-place, and a surgical robotics task. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information.