Phys3DGS: Physically-based 3D Gaussian Splatting for Inverse Rendering
This work addresses rendering quality issues in 3D Gaussian splatting for inverse rendering, which is important for applications like computer graphics and vision, but it appears incremental as it builds on existing methods with specific improvements.
The paper tackled the problem of hidden Gaussians degrading rendering quality in 3D Gaussian splatting-based inverse rendering by proposing deferred rendering and a mesh-based representation, resulting in significantly better rendering quality than existing methods and real-time performance compared to SOTA voxel grid-based approaches.
We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.