Deep Optimized Priors for 3D Shape Modeling and Reconstruction
This work tackles the problem of poor generalization in 3D shape modeling for researchers and practitioners working with limited 3D data, offering an incremental improvement by combining learning and optimization.
This paper addresses the challenge of limited generalization in 3D shape modeling and reconstruction due to sparse 3D datasets. The authors propose a framework that optimizes a pre-trained deep generator's prior and latent code at test time, enabling high-quality adaptation to unseen data from sparse or collapsed observations.
Many learning-based approaches have difficulty scaling to unseen data, as the generality of its learned prior is limited to the scale and variations of the training samples. This holds particularly true with 3D learning tasks, given the sparsity of 3D datasets available. We introduce a new learning framework for 3D modeling and reconstruction that greatly improves the generalization ability of a deep generator. Our approach strives to connect the good ends of both learning-based and optimization-based methods. In particular, unlike the common practice that fixes the pre-trained priors at test time, we propose to further optimize the learned prior and latent code according to the input physical measurements after the training. We show that the proposed strategy effectively breaks the barriers constrained by the pre-trained priors and could lead to high-quality adaptation to unseen data. We realize our framework using the implicit surface representation and validate the efficacy of our approach in a variety of challenging tasks that take highly sparse or collapsed observations as input. Experimental results show that our approach compares favorably with the state-of-the-art methods in terms of both generality and accuracy.