Multi-Spectral Gaussian Splatting with Neural Color Representation
This work addresses the challenge of multi-spectral 3D reconstruction for applications like agriculture, offering a versatile solution without algorithmic changes, though it is incremental in building upon existing 3D Gaussian Splatting frameworks.
The paper tackles the problem of generating multi-view consistent novel views from images of multiple independent cameras with different spectral domains, such as thermal and near-infrared, without requiring cross-modal camera calibration. The result is improved multi-spectral rendering quality and enhanced per-spectra rendering over state-of-the-art methods, as demonstrated in agricultural applications like rendering vegetation indices.
We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).