Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
This addresses the problem of inflexibility in existing EO foundation models for researchers and practitioners dealing with diverse sensor modalities, representing a novel method for a known bottleneck.
The paper tackles the challenge of Earth observation (EO) in open-world settings by proposing the Dynamic One-For-All (DOFA) model, a unified multimodal foundation framework that achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities with reduced computational resources.
Earth observation (EO) in open-world settings presents a unique challenge: different applications rely on diverse sensor modalities, each with varying ground sampling distances, spectral ranges, and numbers of spectral bands. However, existing EO foundation models are typically tailored to specific sensor types, making them inflexible when generalizing across the heterogeneous landscape of EO data. To address this, we propose the Dynamic One-For-All (DOFA) model, a unified, multimodal foundation framework designed for diverse vision tasks in EO. Inspired by neural plasticity, DOFA utilizes a wavelength-conditioned dynamic hypernetwork to process inputs from five distinct satellite sensors flexibly. By continually pretraining on five EO modalities, DOFA achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities. Enhanced with hybrid continual pretraining, DOFA+ requires significantly fewer computational resources while outperforming counterparts trained with extensive GPU budgets. Experiments on diverse datasets highlight DOFA's potential as a foundation for general-purpose vision models in the sensor-diverse EO domain. The code and pre-trained weights are publicly available at https://github.com/zhu-xlab/DOFA.