ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
This addresses the need for efficient and integrated 3D reconstruction for applications in computer graphics and vision, though it is incremental as it builds on existing multi-view and disentanglement methods.
The paper tackles the problem of reconstructing 3D assets from images by presenting ReLi3D, a unified pipeline that simultaneously reconstructs geometry, materials, and illumination from sparse multi-view images in under one second, achieving generalizable results in geometry, material accuracy, and illumination quality.
Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/