GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation
This provides a practical, sensor-agnostic solution for robotic manipulation, addressing the need for precise 3D scene understanding in real-world applications.
The paper tackled the problem of enabling robots to perform manipulation tasks by leveraging multi-view images to infer 3D geometry without depth sensors, resulting in a policy that outperforms state-of-the-art methods on benchmarks and transfers effectively to real-world robots with minimal fine-tuning.
Effective robotic manipulation relies on a precise understanding of 3D scene geometry, and one of the most straightforward ways to acquire such geometry is through multi-view observations. Motivated by this, we present GP3 -- a 3D geometry-aware robotic manipulation policy that leverages multi-view input. GP3 employs a spatial encoder to infer dense spatial features from RGB observations, which enable the estimation of depth and camera parameters, leading to a compact yet expressive 3D scene representation tailored for manipulation. This representation is fused with language instructions and translated into continuous actions via a lightweight policy head. Comprehensive experiments demonstrate that GP3 consistently outperforms state-of-the-art methods on simulated benchmarks. Furthermore, GP3 transfers effectively to real-world robots without depth sensors or pre-mapped environments, requiring only minimal fine-tuning. These results highlight GP3 as a practical, sensor-agnostic solution for geometry-aware robotic manipulation.