MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo
This work addresses the challenge of data acquisition in photometric stereo for applications requiring accurate surface normals, though it is incremental as it builds on existing inverse rendering and relighting methods.
The paper tackles the problem of complex data acquisition in photometric stereo by proposing MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting, achieving high-quality shape, material estimation, and relighting that generalizes well to real-world images.
Photometric stereo typically demands intricate data acquisition setups involving multiple light sources to recover surface normals accurately. In this paper, we propose MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting within a single unified framework. We evaluate the performance of photometric stereo methods using these relit images and demonstrate how they can circumvent the underlying challenge of complex data acquisition. Our physically-based model is trained on a large synthetic dataset containing complex shapes with spatially varying BRDF and is designed to handle indirect illumination effects to improve material reconstruction and relighting. Through extensive qualitative and quantitative evaluation, we demonstrate that the proposed framework generalizes well to real-world images, achieving high-quality shape, material estimation, and relighting. We assess these synthetically relit images over photometric stereo benchmark methods for their physical correctness and resulting normal estimation accuracy, paving the way towards single-shot photometric stereo through physically-based relighting. This work allows us to address the single image-based inverse rendering problem holistically, applying well to both synthetic and real data and taking a step towards mitigating the challenge of data acquisition in photometric stereo.