Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects
This addresses the challenge of accurate 3D reconstruction from limited data for applications in computer vision and graphics, though it is incremental by building on inverse graphics and multi-view cues.
The paper tackles the problem of 3D reconstruction from a single image by introducing Structure from Duplicates (SfD), which uses multiple identical objects in an image as a prior to jointly estimate geometry, material, and illumination, resulting in more realistic and detailed reconstructions that outperform existing single-image and multi-view methods.
Our world is full of identical objects (\emphe.g., cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances.An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations.