Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery
This work addresses the challenge of efficiently applying vision-language models to satellite imagery, particularly for operational systems that often lack full multi-spectral coverage, by enabling RGB-only inference while leveraging spectral information.
The paper introduces SATtxt, a spectrum-aware vision-language foundation model that uses only RGB inputs during inference while retaining spectral cues learned during training. This model improves zero-shot classification by 4.2%, retrieval by 5.9%, and linear probing by 2.7% on average across EuroSAT, BigEarthNet, and ForestNet datasets.
Vision-language foundation models (VLFMs) promise zero-shot and retrieval understanding for Earth observation. While operational satellite systems often lack full multi-spectral coverage, making RGB-only inference highly desirable for scalable deployment, the adoption of VLFMs for satellite imagery remains hindered by two factors: (1) multi-spectral inputs are informative but difficult to exploit consistently due to band redundancy and misalignment; and (2) CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment. We present SATtxt, a spectrum-aware VLFM that operates with RGB inputs only at inference while retaining spectral cues learned during training. Our framework comprises two stages. First, Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector. Second, Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space. Across EuroSAT, BigEarthNet, and ForestNet, SATtxt improves zero-shot classification on average by 4.2%, retrieval by 5.9%, and linear probing by 2.7% over baselines, showing an efficient path toward spectrum-aware vision-language learning for Earth observation. Project page: https://ikhado.github.io/sattxt/