Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping
This work addresses the challenge of data-efficient and interpretable robot autonomy in unseen real-world settings, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of zero-shot scene reconstruction and robot grasping in novel environments by introducing a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering, achieving performance that outperforms existing algorithms on model-free few-shot pose estimation benchmarks.
Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free few-shot benchmarks and demonstrated that it outperforms existing algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive datasets or test-time sampling, our approach offers a pathway towards more data efficient, interpretable and generalizable robot autonomy in novel environments.