GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks
This addresses the problem of inadequate evaluation tools for geospatial applications in AI, such as environmental monitoring and disaster management, but is incremental as it adapts existing benchmarking approaches to a specific domain.
The authors tackled the lack of benchmarks for evaluating vision-language models on geospatial tasks by introducing GEOBench-VLM, a comprehensive dataset with over 10,000 instructions, and found that state-of-the-art models like LLaVa-OneVision achieve only 41.7% accuracy, highlighting significant performance gaps.
While numerous recent benchmarks focus on evaluating generic Vision-Language Models (VLMs), they do not effectively address the specific challenges of geospatial applications. Generic VLM benchmarks are not designed to handle the complexities of geospatial data, an essential component for applications such as environmental monitoring, urban planning, and disaster management. Key challenges in the geospatial domain include temporal change detection, large-scale object counting, tiny object detection, and understanding relationships between entities in remote sensing imagery. To bridge this gap, we present GEOBench-VLM, a comprehensive benchmark specifically designed to evaluate VLMs on geospatial tasks, including scene understanding, object counting, localization, fine-grained categorization, segmentation, and temporal analysis. Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales. We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges. The results indicate that although existing VLMs demonstrate potential, they face challenges when dealing with geospatial-specific tasks, highlighting the room for further improvements. Notably, the best-performing LLaVa-OneVision achieves only 41.7% accuracy on MCQs, slightly more than GPT-4o, which is approximately double the random guess performance. Our benchmark is publicly available at https://github.com/The-AI-Alliance/GEO-Bench-VLM .