3DMiner: Discovering Shapes from Large-Scale Unannotated Image Datasets
This addresses the challenge of obtaining 3D shape and pose annotations from unlabeled images, which is incremental as it builds on self-supervised representations and bundle-adjustment techniques.
The paper tackles the problem of unsupervised 3D shape reconstruction from large-scale unannotated image datasets by proposing 3DMiner, which clusters images with similar shapes and uses correspondences for camera estimation and neural occupancy fields, resulting in significantly better performance than state-of-the-art methods on Pix3D chairs and application to in-the-wild data from LAION-5B.
We present 3DMiner -- a pipeline for mining 3D shapes from challenging large-scale unannotated image datasets. Unlike other unsupervised 3D reconstruction methods, we assume that, within a large-enough dataset, there must exist images of objects with similar shapes but varying backgrounds, textures, and viewpoints. Our approach leverages the recent advances in learning self-supervised image representations to cluster images with geometrically similar shapes and find common image correspondences between them. We then exploit these correspondences to obtain rough camera estimates as initialization for bundle-adjustment. Finally, for every image cluster, we apply a progressive bundle-adjusting reconstruction method to learn a neural occupancy field representing the underlying shape. We show that this procedure is robust to several types of errors introduced in previous steps (e.g., wrong camera poses, images containing dissimilar shapes, etc.), allowing us to obtain shape and pose annotations for images in-the-wild. When using images from Pix3D chairs, our method is capable of producing significantly better results than state-of-the-art unsupervised 3D reconstruction techniques, both quantitatively and qualitatively. Furthermore, we show how 3DMiner can be applied to in-the-wild data by reconstructing shapes present in images from the LAION-5B dataset. Project Page: https://ttchengab.github.io/3dminerOfficial