REOBench: Benchmarking Robustness of Earth Observation Foundation Models
This work addresses the vulnerability of Earth observation models used in critical applications like urban planning and disaster response, providing a benchmark for developing more robust models, but it is incremental as it focuses on evaluation rather than new methods.
The paper tackled the problem of evaluating the robustness of Earth observation foundation models under real-world perturbations by introducing REOBench, a comprehensive benchmark across six tasks and twelve types of image corruptions, revealing significant performance degradation with drops ranging from less than 1% to over 20%.
Earth observation foundation models have shown strong generalization across multiple Earth observation tasks, but their robustness under real-world perturbations remains underexplored. To bridge this gap, we introduce REOBench, the first comprehensive benchmark for evaluating the robustness of Earth observation foundation models across six tasks and twelve types of image corruptions, including both appearance-based and geometric perturbations. To ensure realistic and fine-grained evaluation, our benchmark focuses on high-resolution optical remote sensing images, which are widely used in critical applications such as urban planning and disaster response. We conduct a systematic evaluation of a broad range of models trained using masked image modeling, contrastive learning, and vision-language pre-training paradigms. Our results reveal that (1) existing Earth observation foundation models experience significant performance degradation when exposed to input corruptions. (2) The severity of degradation varies across tasks, model architectures, backbone sizes, and types of corruption, with performance drop varying from less than 1% to over 20%. (3) Vision-language models show enhanced robustness, particularly in multimodal tasks. REOBench underscores the vulnerability of current Earth observation foundation models to real-world corruptions and provides actionable insights for developing more robust and reliable models. Code and data are publicly available at https://github.com/lx709/REOBench.