Progressive Learning of 3D Reconstruction Network from 2D GAN Data
This work addresses the high cost of annotated data for 3D reconstruction, providing a bridge from 2D GAN supervision to 3D models, which is incremental in leveraging generated data but novel in its learning approach.
The paper tackles the problem of reconstructing high-quality textured 3D models from single images by using GAN-generated multi-view datasets to avoid expensive annotations, achieving state-of-the-art results on challenging objects with significant improvements over previous methods.
This paper presents a method to reconstruct high-quality textured 3D models from single images. Current methods rely on datasets with expensive annotations; multi-view images and their camera parameters. Our method relies on GAN generated multi-view image datasets which have a negligible annotation cost. However, they are not strictly multi-view consistent and sometimes GANs output distorted images. This results in degraded reconstruction qualities. In this work, to overcome these limitations of generated datasets, we have two main contributions which lead us to achieve state-of-the-art results on challenging objects: 1) A robust multi-stage learning scheme that gradually relies more on the models own predictions when calculating losses, 2) A novel adversarial learning pipeline with online pseudo-ground truth generations to achieve fine details. Our work provides a bridge from 2D supervisions of GAN models to 3D reconstruction models and removes the expensive annotation efforts. We show significant improvements over previous methods whether they were trained on GAN generated multi-view images or on real images with expensive annotations. Please visit our web-page for 3D visuals: https://research.nvidia.com/labs/adlr/progressive-3d-learning