SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation
This addresses data-efficient and generalizable robotic manipulation for real-world applications like garment handling and table rearrangement, representing a novel method rather than incremental progress.
The paper tackles the problem of high sample complexity in imitation learning for complex robotic tasks by proposing Semantic Keypoint Imitation Learning (SKIL), which uses vision foundation models to extract semantic keypoints, achieving double the performance of baselines in some tasks and a 70% success rate in long-horizon tasks with only 30 demonstrations.
Real-world tasks such as garment manipulation and table rearrangement demand robots to perform generalizable, highly precise, and long-horizon actions. Although imitation learning has proven to be an effective approach for teaching robots new skills, large amounts of expert demonstration data are still indispensible for these complex tasks, resulting in high sample complexity and costly data collection. To address this, we propose Semantic Keypoint Imitation Learning (SKIL), a framework which automatically obtains semantic keypoints with the help of vision foundation models, and forms the descriptor of semantic keypoints that enables efficient imitation learning of complex robotic tasks with significantly lower sample complexity. In real-world experiments, SKIL doubles the performance of baseline methods in tasks such as picking a cup or mouse, while demonstrating exceptional robustness to variations in objects, environmental changes, and distractors. For long-horizon tasks like hanging a towel on a rack where previous methods fail completely, SKIL achieves a mean success rate of 70\% with as few as 30 demonstrations. Furthermore, SKIL naturally supports cross-embodiment learning due to its semantic keypoints abstraction. Our experiments demonstrate that even human videos bring considerable improvement to the learning performance. All these results demonstrate the great success of SKIL in achieving data-efficient generalizable robotic learning. Visualizations and code are available at: https://skil-robotics.github.io/SKIL-robotics/.