GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields
This addresses the challenge of multi-task robotic manipulation with strong generalization, which is crucial for deploying robots in real-world settings, though it appears incremental by building on existing neural field and foundation model techniques.
The paper tackles the problem of enabling robots to perform diverse manipulation tasks from visual observations in unstructured real-world environments by developing GNFactor, a visual behavior cloning agent that integrates generalizable neural feature fields with a Perceiver Transformer, achieving substantial improvements over state-of-the-art methods in seen and unseen tasks.
It is a long-standing problem in robotics to develop agents capable of executing diverse manipulation tasks from visual observations in unstructured real-world environments. To achieve this goal, the robot needs to have a comprehensive understanding of the 3D structure and semantics of the scene. In this work, we present $\textbf{GNFactor}$, a visual behavior cloning agent for multi-task robotic manipulation with $\textbf{G}$eneralizable $\textbf{N}$eural feature $\textbf{F}$ields. GNFactor jointly optimizes a generalizable neural field (GNF) as a reconstruction module and a Perceiver Transformer as a decision-making module, leveraging a shared deep 3D voxel representation. To incorporate semantics in 3D, the reconstruction module utilizes a vision-language foundation model ($\textit{e.g.}$, Stable Diffusion) to distill rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3 real robot tasks and perform detailed ablations on 10 RLBench tasks with a limited number of demonstrations. We observe a substantial improvement of GNFactor over current state-of-the-art methods in seen and unseen tasks, demonstrating the strong generalization ability of GNFactor. Our project website is https://yanjieze.com/GNFactor/ .