Simple and Effective Synthesis of Indoor 3D Scenes
This addresses the problem of complex and multi-stage 3D scene synthesis for applications in robotics and embodied AI, offering a simpler and more effective alternative.
The paper tackles the problem of synthesizing immersive 3D indoor scenes from images by proposing a simple image-to-image GAN that maps from reprojections of incomplete point clouds to full high-resolution RGB-D images. It significantly outperforms prior work on Matterport3D and RealEstate10K datasets in human evaluations and FID scores, and improves a VLN agent's success rate by up to 1.5% on the R2R benchmark through generative data augmentation.
We study the problem of synthesizing immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints, including viewpoints that extrapolate far beyond the input images while maintaining 3D consistency. Existing approaches are highly complex, with many separately trained stages and components. We propose a simple alternative: an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images. On the Matterport3D and RealEstate10K datasets, our approach significantly outperforms prior work when evaluated by humans, as well as on FID scores. Further, we show that our model is useful for generative data augmentation. A vision-and-language navigation (VLN) agent trained with trajectories spatially-perturbed by our model improves success rate by up to 1.5% over a state of the art baseline on the R2R benchmark. Our code will be made available to facilitate generative data augmentation and applications to downstream robotics and embodied AI tasks.