Weakly supervised 3D Reconstruction with Adversarial Constraint
This work addresses the annotation cost problem for 3D reconstruction in computer vision, offering an incremental improvement by leveraging weak supervision and adversarial constraints.
The paper tackles the problem of expensive 3D CAD annotations for supervised 3D reconstruction by using inexpensive 2D foreground masks as weak supervision, proposing a method that constrains reconstructions to a manifold of realistic 3D shapes and demonstrates competitive performance on synthetic and real datasets.
Supervised 3D reconstruction has witnessed a significant progress through the use of deep neural networks. However, this increase in performance requires large scale annotations of 2D/3D data. In this paper, we explore inexpensive 2D supervision as an alternative for expensive 3D CAD annotation. Specifically, we use foreground masks as weak supervision through a raytrace pooling layer that enables perspective projection and backpropagation. Additionally, since the 3D reconstruction from masks is an ill posed problem, we propose to constrain the 3D reconstruction to the manifold of unlabeled realistic 3D shapes that match mask observations. We demonstrate that learning a log-barrier solution to this constrained optimization problem resembles the GAN objective, enabling the use of existing tools for training GANs. We evaluate and analyze the manifold constrained reconstruction on various datasets for single and multi-view reconstruction of both synthetic and real images.