Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes
This addresses a challenging domain-specific problem in computer vision for applications like robotics or AR/VR, with incremental improvements in method design.
The paper tackles the ill-posed problem of 3D reconstruction of transparent objects from a few unconstrained natural images, achieving successful recovery of high-quality 3D geometry using as few as 5-12 images.
Recovering the 3D shape of transparent objects using a small number of unconstrained natural images is an ill-posed problem. Complex light paths induced by refraction and reflection have prevented both traditional and deep multiview stereo from solving this challenge. We propose a physically-based network to recover 3D shape of transparent objects using a few images acquired with a mobile phone camera, under a known but arbitrary environment map. Our novel contributions include a normal representation that enables the network to model complex light transport through local computation, a rendering layer that models refractions and reflections, a cost volume specifically designed for normal refinement of transparent shapes and a feature mapping based on predicted normals for 3D point cloud reconstruction. We render a synthetic dataset to encourage the model to learn refractive light transport across different views. Our experiments show successful recovery of high-quality 3D geometry for complex transparent shapes using as few as 5-12 natural images. Code and data are publicly released.