CPopQA: Ranking Cultural Concept Popularity by LLMs
This work addresses the underexplored issue of how well LLMs capture corpus-level statistical trends for reasoning, particularly for long-tail concepts, which is incremental in evaluating LLM knowledge capacity.
The paper tackles the problem of assessing large language models' (LLMs) ability to rank the popularity of long-tail cultural concepts, such as holidays, in the US and UK, by introducing a few-shot QA task (CPopQA) with 6,000 testing pairs, and finds that large models like GPT-3.5 perform well and can identify geo-cultural proximity.
Prior work has demonstrated large language models' (LLMs) potential to discern statistical tendencies within their pre-training corpora. Despite that, many examinations of LLMs' knowledge capacity focus on knowledge explicitly appearing in the training data or implicitly inferable from similar contexts. How well an LLM captures the corpus-level statistical trends of concepts for reasoning, especially long-tail ones, is still underexplored. In this study, we introduce a novel few-shot question-answering task (CPopQA) that examines LLMs' statistical ranking abilities for long-tail cultural concepts (e.g., holidays), with a specific focus on these concepts' popularity in the United States and the United Kingdom, respectively. We curate a dataset containing 459 holidays across 58 countries, generating a total of 6,000 QA testing pairs. Experiments on four strong LLMs show that large models are capable of ranking long-tail cultural concepts regarding their statistical tendency. Notably, GPT-3.5 displayed superior performance and exhibited its potential to identify geo-cultural proximity across continents.