OBSR: Open Benchmark for Spatial Representations
This addresses the problem of limited progress in GeoAI for researchers and practitioners by providing a foundational benchmark, though it is incremental as it builds on existing data and tasks.
The paper tackles the lack of standardized benchmarks in GeoAI by introducing OBSR, a multi-task, modality-agnostic benchmark with 7 datasets from diverse cities, establishing baselines for evaluating geospatial embedders.
GeoAI is evolving rapidly, fueled by diverse geospatial datasets like traffic patterns, environmental data, and crowdsourced OpenStreetMap (OSM) information. While sophisticated AI models are being developed, existing benchmarks are often concentrated on single tasks and restricted to a single modality. As such, progress in GeoAI is limited by the lack of a standardized, multi-task, modality-agnostic benchmark for their systematic evaluation. This paper introduces a novel benchmark designed to assess the performance, accuracy, and efficiency of geospatial embedders. Our benchmark is modality-agnostic and comprises 7 distinct datasets from diverse cities across three continents, ensuring generalizability and mitigating demographic biases. It allows for the evaluation of GeoAI embedders on various phenomena that exhibit underlying geographic processes. Furthermore, we establish a simple and intuitive task-oriented model baselines, providing a crucial reference point for comparing more complex solutions.