FLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing
This addresses a specific bottleneck in remote sensing AI by improving multimodal pretraining for better vision-only tasks, though it is incremental as it builds on existing methods like CLIP and MAE.
The paper tackles the problem of degraded vision-only performance in remote sensing when using contrastive image-text pretraining like CLIP, proposing FLAVARS, which combines contrastive learning and masked modeling with geospatial alignment, resulting in a +6% mIOU improvement on SpaceNet1 for semantic segmentation while retaining zero-shot classification ability.
Remote sensing imagery is dense with objects and contextual visual information. There is a recent trend to combine paired satellite images and text captions for pretraining performant encoders for downstream tasks. However, while contrastive image-text methods like CLIP enable vision-language alignment and zero-shot classification ability, vision-only downstream performance tends to degrade compared to image-only pretraining, such as MAE. In this paper, we propose FLAVARS, a pretraining method that combines the best of both contrastive learning and masked modeling, along with geospatial alignment via contrastive location encoding. We find that FLAVARS significantly outperforms a baseline of SkyCLIP for vision-only tasks such as KNN classification and semantic segmentation, +6\% mIOU on SpaceNet1, while retaining the ability to perform zero-shot classification, unlike MAE pretrained methods.