Monocular Neural Image Based Rendering with Continuous View Control
This addresses the challenge of generating realistic 3D views from limited 2D data for applications in computer vision and graphics, representing an incremental improvement by combining existing techniques like transforming auto-encoders and depth-guided warping.
The paper tackles the problem of synthesizing high-quality novel views of 3D objects or scenes from monocular images with precise 6-DOF viewpoint control, achieving results that preserve details and generalize to unseen images from the internet.
We present an approach that learns to synthesize high-quality, novel views of 3D objects or scenes, while providing fine-grained and precise control over the 6-DOF viewpoint. The approach is self-supervised and only requires 2D images and associated view transforms for training. Our main contribution is a network architecture that leverages a transforming auto-encoder in combination with a depth-guided warping procedure to predict geometrically accurate unseen views. Leveraging geometric constraints renders direct supervision via depth or flow maps unnecessary. If large parts of the object are occluded in the source view, a purely learning based prior is used to predict the values for dis-occluded pixels. Our network furthermore predicts a per-pixel mask, used to fuse depth-guided and pixel-based predictions. The resulting images reflect the desired 6-DOF transformation and details are preserved. We thoroughly evaluate our architecture on synthetic and real scenes and under fine-grained and fixed-view settings. Finally, we demonstrate that the approach generalizes to entirely unseen images such as product images downloaded from the internet.