P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies
This work addresses the problem of brittle robot manipulation policies that fail to transfer beyond training data, offering a domain-specific solution for robotics.
The paper tackles the challenge of developing generalizable robot policies that handle varied environmental conditions and object instances by introducing P3-PO, a framework that uses prescriptive point priors to create a state representation, resulting in a 43% absolute improvement over prior methods in real-world tasks.
Developing generalizable robot policies that can robustly handle varied environmental conditions and object instances remains a fundamental challenge in robot learning. While considerable efforts have focused on collecting large robot datasets and developing policy architectures to learn from such data, naively learning from visual inputs often results in brittle policies that fail to transfer beyond the training data. This work presents Prescriptive Point Priors for Policies or P3-PO, a novel framework that constructs a unique state representation of the environment leveraging recent advances in computer vision and robot learning to achieve improved out-of-distribution generalization for robot manipulation. This representation is obtained through two steps. First, a human annotator prescribes a set of semantically meaningful points on a single demonstration frame. These points are then propagated through the dataset using off-the-shelf vision models. The derived points serve as an input to state-of-the-art policy architectures for policy learning. Our experiments across four real-world tasks demonstrate an overall 43% absolute improvement over prior methods when evaluated in identical settings as training. Further, P3-PO exhibits 58% and 80% gains across tasks for new object instances and more cluttered environments respectively. Videos illustrating the robot's performance are best viewed at point-priors.github.io.