Towards Learning a Generalizable 3D Scene Representation from 2D Observations
This enables robotic manipulation by generalizing to unseen object arrangements, though it is incremental as it builds on prior neural radiance field methods.
The paper tackles the problem of predicting 3D workspace occupancy from 2D robot observations by introducing a Generalizable Neural Radiance Field that operates in a global frame, achieving a 26mm reconstruction error on real scenes without scene-specific finetuning.
We introduce a Generalizable Neural Radiance Field approach for predicting 3D workspace occupancy from egocentric robot observations. Unlike prior methods operating in camera-centric coordinates, our model constructs occupancy representations in a global workspace frame, making it directly applicable to robotic manipulation. The model integrates flexible source views and generalizes to unseen object arrangements without scene-specific finetuning. We demonstrate the approach on a humanoid robot and evaluate predicted geometry against 3D sensor ground truth. Trained on 40 real scenes, our model achieves 26mm reconstruction error, including occluded regions, validating its ability to infer complete 3D occupancy beyond traditional stereo vision methods.