Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery
This work addresses data-efficiency and generalization challenges for researchers and practitioners in satellite machine learning, though it is incremental as it builds on existing multi-modal fusion concepts in a specific domain.
The study tackled the problem of limited data-efficiency and poor out-of-distribution generalization in machine learning with satellite imagery by fusing additional geographic data layers with optical imagery, finding that this approach significantly improves model performance, especially in data-scarce and out-of-sample settings, with hard-coded fusion strategies outperforming learned ones.
A large variety of geospatial data layers is available around the world ranging from remotely-sensed raster data like satellite imagery, digital elevation models, predicted land cover maps, and human-annotated data, to data derived from environmental sensors such as air temperature or wind speed data. A large majority of machine learning models trained on satellite imagery (SatML), however, are designed primarily for optical input modalities such as multi-spectral satellite imagery. To better understand the value of using other input modalities alongside optical imagery in supervised learning settings, we generate augmented versions of SatML benchmark tasks by appending additional geographic data layers to datasets spanning classification, regression, and segmentation. Using these augmented datasets, we find that fusing additional geographic inputs with optical imagery can significantly improve SatML model performance. Benefits are largest in settings where labeled data are limited and in geographic out-of-sample settings, suggesting that multi-modal inputs may be especially valuable for data-efficiency and out-of-sample performance of SatML models. Surprisingly, we find that hard-coded fusion strategies outperform learned variants, with interesting implications for future work.