DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting
This addresses a challenging problem in computer vision for applications like 3D scanning in uncontrolled environments, though it is an incremental improvement over existing methods.
The paper tackles geometry reconstruction of textureless, non-Lambertian objects under unknown natural illumination by introducing DeepShaRM, a multi-view method that recovers shape and reflectance maps without disentangling reflectance and illumination, achieving state-of-the-art accuracy on synthetic and real-world data.
Geometry reconstruction of textureless, non-Lambertian objects under unknown natural illumination (i.e., in the wild) remains challenging as correspondences cannot be established and the reflectance cannot be expressed in simple analytical forms. We derive a novel multi-view method, DeepShaRM, that achieves state-of-the-art accuracy on this challenging task. Unlike past methods that formulate this as inverse-rendering, i.e., estimation of reflectance, illumination, and geometry from images, our key idea is to realize that reflectance and illumination need not be disentangled and instead estimated as a compound reflectance map. We introduce a novel deep reflectance map estimation network that recovers the camera-view reflectance maps from the surface normals of the current geometry estimate and the input multi-view images. The network also explicitly estimates per-pixel confidence scores to handle global light transport effects. A deep shape-from-shading network then updates the geometry estimate expressed with a signed distance function using the recovered reflectance maps. By alternating between these two, and, most important, by bypassing the ill-posed problem of reflectance and illumination decomposition, the method accurately recovers object geometry in these challenging settings. Extensive experiments on both synthetic and real-world data clearly demonstrate its state-of-the-art accuracy.