What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction
This work solves the problem of accurate 3D reconstruction from single images for applications in computer vision and robotics, representing a novel method rather than an incremental improvement.
The paper tackles the problem of full 3D scene reconstruction from a single unseen image by proposing the Directed Ray Distance Function (DRDF), which addresses challenges in predicting image-conditioned distance functions, and shows that it outperforms other methods on datasets like Matterport3D, 3DFront, and ScanNet.
We present an approach for full 3D scene reconstruction from a single unseen image. We train on dataset of realistic non-watertight scans of scenes. Our approach predicts a distance function, since these have shown promise in handling complex topologies and large spaces. We identify and analyze two key challenges for predicting such image conditioned distance functions that have prevented their success on real 3D scene data. First, we show that predicting a conventional scene distance from an image requires reasoning over a large receptive field. Second, we analytically show that the optimal output of the network trained to predict these distance functions does not obey all the distance function properties. We propose an alternate distance function, the Directed Ray Distance Function (DRDF), that tackles both challenges. We show that a deep network trained to predict DRDFs outperforms all other methods quantitatively and qualitatively on 3D reconstruction from single image on Matterport3D, 3DFront, and ScanNet.