PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
This addresses generalization issues in robot manipulation for robotics researchers, offering a scalable solution with significant performance boosts.
The paper tackles the problem of robotic manipulation policies failing to generalize by offloading high-level reasoning to vision-language models (VLMs) to predict point-based representations for actions and focus areas, resulting in a 41.4x improvement in zero-shot generalization for a 3D policy trained in simulation and 2-3.5x gains for various policies.
Robotic manipulation policies often fail to generalize because they must simultaneously learn where to attend, what actions to take, and how to execute them. We argue that high-level reasoning about where and what can be offloaded to vision-language models (VLMs), leaving policies to specialize in how to act. We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which fine-tunes VLMs to predict a unified point-based intermediate representation: 1. end-effector paths specifying what actions to take, and 2. task-relevant masks indicating where to focus. These annotations are directly overlaid onto robot observations, making the representation policy-agnostic and transferable across architectures. To enable scalable training, we introduce an automatic annotation pipeline, generating labeled data across 20+ robot datasets spanning 9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot generalization, including a 41.4x real-world improvement for a 3D policy trained only in simulation, and 2-3.5x gains for both large VLAs and small manipulation policies. By letting VLMs absorb semantic and visual complexity, PEEK equips manipulation policies with the minimal cues they need--where, what, and how. Website at https://peek-robot.github.io/.