GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation
This work addresses the need for rigorous evaluation of LLMs in GIS automation for researchers and practitioners, providing a reproducible benchmark to advance GeoAI with human-in-the-loop support, though it is incremental as it focuses on assessment rather than new methods.
The authors tackled the problem of uncertain capabilities of large language models (LLMs) in automating geospatial analysis by creating GeoAnalystBench, a benchmark of 50 Python-based tasks, and found that proprietary models like ChatGPT-4o-mini achieved high validity (95%) and code alignment (CodeBLEU 0.39), while smaller open-source models like DeepSeek-R1-7B performed poorly (48.5% validity, CodeBLEU 0.272).
Recent advances in large language models (LLMs) have fueled growing interest in automating geospatial analysis and GIS workflows, yet their actual capabilities remain uncertain. In this work, we call for rigorous evaluation of LLMs on well-defined geoprocessing tasks before making claims about full GIS automation. To this end, we present GeoAnalystBench, a benchmark of 50 Python-based tasks derived from real-world geospatial problems and carefully validated by GIS experts. Each task is paired with a minimum deliverable product, and evaluation covers workflow validity, structural alignment, semantic similarity, and code quality (CodeBLEU). Using this benchmark, we assess both proprietary and open source models. Results reveal a clear gap: proprietary models such as ChatGPT-4o-mini achieve high validity 95% and stronger code alignment (CodeBLEU 0.39), while smaller open source models like DeepSeek-R1-7B often generate incomplete or inconsistent workflows (48.5% validity, 0.272 CodeBLEU). Tasks requiring deeper spatial reasoning, such as spatial relationship detection or optimal site selection, remain the most challenging across all models. These findings demonstrate both the promise and limitations of current LLMs in GIS automation and provide a reproducible framework to advance GeoAI research with human-in-the-loop support.