Quantifying Geospatial in the Common Crawl Corpus
This work addresses the lack of quantitative insights into geospatial data in Common Crawl, impacting researchers studying LLM biases, but it is incremental as it primarily provides foundational measurements without introducing new methods or broad SOTA results.
The paper tackled the problem of quantifying geospatial content in the Common Crawl corpus, which is crucial for understanding LLMs' spatial reasoning, and found that 18.7% of web documents contain geospatial information like coordinates and addresses, with little difference between English and non-English documents.
Large language models (LLMs) exhibit emerging geospatial capabilities, stemming from their pre-training on vast unlabelled text datasets that are often derived from the Common Crawl (CC) corpus. However, the geospatial content within CC remains largely unexplored, impacting our understanding of LLMs' spatial reasoning. This paper investigates the prevalence of geospatial data in recent Common Crawl releases using Gemini 1.5, a powerful language model. By analyzing a sample of documents and manually revising the results, we estimate that 18.7% of web documents in CC contain geospatial information such as coordinates and addresses. We find little difference in prevalence between Enlgish- and non-English-language documents. Our findings provide quantitative insights into the nature and extent of geospatial data in CC, and lay the groundwork for future studies of geospatial biases of LLMs.