GPT4GEO: How a Language Model Sees the World's Geography
This work evaluates LLM geographic understanding for applications like geospatial analysis and disaster response, but it is incremental as it focuses on characterizing an existing model.
The researchers assessed GPT-4's factual geographic knowledge and interpretative reasoning abilities through experiments like location estimation and route finding, revealing both surprising capabilities and limitations.
Large language models (LLMs) have shown remarkable capabilities across a broad range of tasks involving question answering and the generation of coherent text and code. Comprehensively understanding the strengths and weaknesses of LLMs is beneficial for safety, downstream applications and improving performance. In this work, we investigate the degree to which GPT-4 has acquired factual geographic knowledge and is capable of using this knowledge for interpretative reasoning, which is especially important for applications that involve geographic data, such as geospatial analysis, supply chain management, and disaster response. To this end, we design and conduct a series of diverse experiments, starting from factual tasks such as location, distance and elevation estimation to more complex questions such as generating country outlines and travel networks, route finding under constraints and supply chain analysis. We provide a broad characterisation of what GPT-4 (without plugins or Internet access) knows about the world, highlighting both potentially surprising capabilities but also limitations.