Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances
This addresses the challenge of robotic manipulation in unstructured environments for robotics and AI applications, representing an incremental advance through integration of existing methods.
The paper tackles the problem of enabling robots to perform long-horizon manipulation tasks on unknown objects without prior knowledge, by integrating task-and-motion planning with perception modules, and demonstrates that a single system can generalize across various objects and tasks without retraining.
We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable objects, their geometries, and their affordances. We explore several different ways of implementing such perceptual modules for segmentation, property detection, shape estimation, and grasp generation. We show how these modules are integrated within the PDDLStream task and motion planning framework. Finally, we demonstrate that this strategy can enable a single system to perform a wide variety of real-world multi-step manipulation tasks, generalizing over a broad class of objects, object arrangements, and goals, without any prior knowledge of the environment and without re-training.