Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors
For practitioners deploying remote sensing models, this work provides a practical solution to adapt to evolving satellite sensors without costly re-labeling and retraining.
This paper addresses the problem of adapting remote sensing models to changes in sensor modalities (substitution, addition, or removal) without full retraining. The proposed DeluluNet architecture achieves effective modality transfer, addition, and peeking using unlabeled multimodal data and modality hallucination, enabling models to handle changing input modalities with minimal retraining.
Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.