ReCap: Better Gaussian Relighting with Cross-Environment Captures
This work improves realistic virtual object placement for applications like AR/VR, but it is incremental as it builds on existing methods with a novel multi-task approach.
The paper tackles the problem of accurate 3D object relighting in unseen environments by addressing albedo-lighting ambiguity, resulting in ReCap outperforming leading competitors on an expanded relighting benchmark.
Accurate 3D objects relighting in diverse unseen environments is crucial for realistic virtual object placement. Due to the albedo-lighting ambiguity, existing methods often fall short in producing faithful relights. Without proper constraints, observed training views can be explained by numerous combinations of lighting and material attributes, lacking physical correspondence with the actual environment maps used for relighting. In this work, we present ReCap, treating cross-environment captures as multi-task target to provide the missing supervision that cuts through the entanglement. Specifically, ReCap jointly optimizes multiple lighting representations that share a common set of material attributes. This naturally harmonizes a coherent set of lighting representations around the mutual material attributes, exploiting commonalities and differences across varied object appearances. Such coherence enables physically sound lighting reconstruction and robust material estimation - both essential for accurate relighting. Together with a streamlined shading function and effective post-processing, ReCap outperforms all leading competitors on an expanded relighting benchmark.