SpecGaussian with Latent Features: A High-quality Modeling of the View-dependent Appearance for 3D Gaussian Splatting
This work addresses limitations in 3D-GS for rendering view-dependent effects like specular reflections, which is important for realistic 3D scene reconstruction in computer vision and graphics applications, though it appears incremental as it builds directly on 3D-GS.
The paper tackles the challenge of modeling specular reflections and anisotropic appearance in 3D Gaussian Splatting (3D-GS) for novel view synthesis, introducing Lantent-SpecGS which uses latent neural descriptors and parallel CNNs to achieve competitive performance in handling complex lighting scenarios.
Recently, the 3D Gaussian Splatting (3D-GS) method has achieved great success in novel view synthesis, providing real-time rendering while ensuring high-quality rendering results. However, this method faces challenges in modeling specular reflections and handling anisotropic appearance components, especially in dealing with view-dependent color under complex lighting conditions. Additionally, 3D-GS uses spherical harmonic to learn the color representation, which has limited ability to represent complex scenes. To overcome these challenges, we introduce Lantent-SpecGS, an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. This enables a more effective representation of 3D feature fields, including appearance and geometry. Moreover, two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image. Experimental results demonstrate that our method obtains competitive performance in novel view synthesis and extends the ability of 3D-GS to handle intricate scenarios with specular reflections.