nLMVS-Net: Deep Non-Lambertian Multi-View Stereo
This addresses the challenge of 3D reconstruction for textureless, complex non-Lambertian surfaces in natural settings, representing a novel method for a known bottleneck in multi-view stereo.
The paper tackles the problem of reconstructing 3D shape and reflectance from multi-view images of non-Lambertian surfaces under natural illumination, achieving robust and accurate recovery as demonstrated on new synthetic and real-world datasets.
We introduce a novel multi-view stereo (MVS) method that can simultaneously recover not just per-pixel depth but also surface normals, together with the reflectance of textureless, complex non-Lambertian surfaces captured under known but natural illumination. Our key idea is to formulate MVS as an end-to-end learnable network, which we refer to as nLMVS-Net, that seamlessly integrates radiometric cues to leverage surface normals as view-independent surface features for learned cost volume construction and filtering. It first estimates surface normals as pixel-wise probability densities for each view with a novel shape-from-shading network. These per-pixel surface normal densities and the input multi-view images are then input to a novel cost volume filtering network that learns to recover per-pixel depth and surface normal. The reflectance is also explicitly estimated by alternating with geometry reconstruction. Extensive quantitative evaluations on newly established synthetic and real-world datasets show that nLMVS-Net can robustly and accurately recover the shape and reflectance of complex objects in natural settings.