PhyGaP: Physically-Grounded Gaussians with Polarization Cues
This addresses the limitation of existing methods in supporting high-fidelity relighting for reflective 3D objects, which is incremental as it builds on 3DGS with new cues.
The paper tackled the problem of reconstructing physical attributes like albedo and reflectance for high-fidelity relighting in 3D Gaussian Splatting by leveraging polarization cues, achieving ~2 dB higher PSNR and 45.7% better Cosine Distance compared to existing RGB-based methods.
Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated great success in modeling reflective 3D objects and their interaction with the environment via deferred rendering (DR). However, existing methods often struggle with correctly reconstructing physical attributes such as albedo and reflectance, and therefore they do not support high-fidelity relighting. Observing that this limitation stems from the lack of shape and material information in RGB images, we present PhyGaP, a physically-grounded 3DGS method that leverages polarization cues to facilitate precise reflection decomposition and visually consistent relighting of reconstructed objects. Specifically, we design a polarimetric deferred rendering (PolarDR) process to model polarization by reflection, and a self-occlusion-aware environment map building technique (GridMap) to resolve indirect lighting of non-convex objects. We validate on multiple synthetic and real-world scenes, including those featuring only partial polarization cues, that PhyGaP not only excels in reconstructing the appearance and surface normal of reflective 3D objects (~2 dB in PSNR and 45.7% in Cosine Distance better than existing RGB-based methods on average), but also achieves state-of-the-art inverse rendering and relighting capability. Our code will be released soon.