GeoArena: An Open Platform for Benchmarking Large Vision-language Models on WorldWide Image Geolocalization
This work provides a more accurate and privacy-conscious evaluation framework for researchers and developers working on image geolocalization tasks, though it is incremental as it focuses on improving benchmarking rather than introducing new geolocalization methods.
The paper tackles the problem of evaluating large vision-language models (LVLMs) on image geolocalization by addressing data leakage and privacy concerns in existing metrics, resulting in the development of GeoArena, an open platform that collected over thousands of voting records in two months to establish a leaderboard for benchmarking.
Image geolocalization aims to predict the geographic location of images captured anywhere on Earth, but its global nature presents significant challenges. Current evaluation methodologies suffer from two major limitations. First, data leakage: advanced approaches often rely on large vision-language models (LVLMs) to predict image locations, yet these models are frequently pretrained on the test datasets, compromising the accuracy of evaluating a model's actual geolocalization capability. Second, existing metrics primarily rely on exact geographic coordinates to assess predictions, which not only neglects the reasoning process but also raises privacy concerns when user-level location data is required. To address these issues, we propose GeoArena, a first open platform for evaluating LVLMs on worldwide image geolocalization tasks, offering true in-the-wild and human-centered benchmarking. GeoArena enables users to upload in-the-wild images for a more diverse evaluation corpus, and it leverages pairwise human judgments to determine which model output better aligns with human expectations. Our platform has been deployed online for two months, during which we collected over thousands voting records. Based on this data, we conduct a detailed analysis and establish a leaderboard of different LVLMs on the image geolocalization task. GeoArena has been open-sourced to support future research.