Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition
For computer graphics and vision applications like relighting and AR, this work demonstrates that object motion, typically a challenge, can be leveraged to resolve the ill-posed material-lighting ambiguity.
The paper shows that observing rigidly moving objects improves material-lighting decomposition in inverse rendering, achieving significantly more accurate material reconstruction compared to static observations, and validates this on both synthetic and real-world data.
Decomposing outgoing surface radiance into material and illumination during inverse rendering is essential for applications such as relighting and augmented reality, yet it is severely ill-posed since multiple combinations can result in the same observed colour. Capturing an object under multiple lighting conditions usually helps resolve this ambiguity as it constrains the optimization towards correct solutions. In this work, we explore the potential of reconstructing rigidly moving objects -- which provides observations of diverse light-surface interactions -- to resolve the material-lighting ambiguity in inverse rendering. For this purpose, we introduce a relightable approach that marries object tracking and reconstruction with inverse rendering for general rigidly moving objects. Our experimental analysis on synthetic data demonstrates that motion can be an advantage for disentangling material and lighting: the reconstructed material is significantly more accurate when the object is observed under rigid motion than when it is static. Moreover, results on RGB videos of real hand-held objects show that our pipeline preserves this advantage even under noisy real-world conditions.