Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization
This work addresses the challenge of adapting MLLMs to out-of-distribution domains like remote sensing for object localization, offering practical guidance for researchers and practitioners in geospatial AI.
The paper analyzes recent multimodal large language models (MLLMs) with fine-grained spatial reasoning capabilities for object localization in earth observation imagery, finding they perform well in zero-shot scenarios and providing optimization insights.
Multimodal large language models (MLLMs) have altered the landscape of computer vision, obtaining impressive results across a wide range of tasks, especially in zero-shot settings. Unfortunately, their strong performance does not always transfer to out-of-distribution domains, such as earth observation (EO) imagery. Prior work has demonstrated that MLLMs excel at some EO tasks, such as image captioning and scene understanding, while failing at tasks that require more fine-grained spatial reasoning, such as object localization. However, MLLMs are advancing rapidly and insights quickly become out-dated. In this work, we analyze more recent MLLMs that have been explicitly trained to include fine-grained spatial reasoning capabilities, benchmarking them on EO object localization tasks. We demonstrate that these models are performant in certain settings, making them well suited for zero-shot scenarios. Additionally, we provide a detailed discussion focused on prompt selection, ground sample distance (GSD) optimization, and analyzing failure cases. We hope that this work will prove valuable as others evaluate whether an MLLM is well suited for a given EO localization task and how to optimize it.