Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
This addresses the problem of 3D reconstruction for computer vision applications, offering a novel training approach that is incremental in improving differentiable renderer smoothness.
The paper tackles 3D geometry reconstruction from images by introducing Pix2Vex, a network that uses a smooth differentiable renderer and an image-to-image translation component to train with minimal supervision, achieving results without needing 3D ground truth.
The long-coveted task of reconstructing 3D geometry from images is still a standing problem. In this paper, we build on the power of neural networks and introduce Pix2Vex, a network trained to convert camera-captured images into 3D geometry. We present a novel differentiable renderer ($DR$) as a forward validation means during training. Our key insight is that $DR$s produce images of a particular appearance, different from typical input images. Hence, we propose adding an image-to-image translation component, converting between these rendering styles. This translation closes the training loop, while allowing to use minimal supervision only, without needing any 3D model as ground truth. Unlike state-of-the-art methods, our $DR$ is $C^\infty$ smooth and thus does not display any discontinuities at occlusions or dis-occlusions. Through our novel training scheme, our network can train on different types of images, where previous work can typically only train on images of a similar appearance to those rendered by a $DR$.