A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Features
This work addresses scene representation problems for remote sensing applications, offering incremental improvements by refining existing methods for multispectral data integration.
The paper tackles the challenge of accurate scene understanding in remote sensing by introducing Spectral LLaVA, a vision-language framework that integrates multispectral data with language alignment, resulting in improved scene classification and detailed descriptions, especially in complex environments like land use areas or coastal regions with snow, clouds, or haze.
Scene understanding in remote sensing often faces challenges in generating accurate representations for complex environments such as various land use areas or coastal regions, which may also include snow, clouds, or haze. To address this, we present a vision-language framework named Spectral LLaVA, which integrates multispectral data with vision-language alignment techniques to enhance scene representation and description. Using the BigEarthNet v2 dataset from Sentinel-2, we establish a baseline with RGB-based scene descriptions and further demonstrate substantial improvements through the incorporation of multispectral information. Our framework optimizes a lightweight linear projection layer for alignment while keeping the vision backbone of SpectralGPT frozen. Our experiments encompass scene classification using linear probing and language modeling for jointly performing scene classification and description generation. Our results highlight Spectral LLaVA's ability to produce detailed and accurate descriptions, particularly for scenarios where RGB data alone proves inadequate, while also enhancing classification performance by refining SpectralGPT features into semantically meaningful representations.