CARTE: A Benchmark for Mapping Language Model Knowledge Across France
For researchers and developers of LLMs, this benchmark highlights the lack of intra-country cultural knowledge in current models, which is an incremental contribution to cultural evaluation benchmarks.
CARTE is a benchmark of 2,431 multiple-choice questions across 13 French regions and 14 domains, used to evaluate LLMs' fine-grained regional knowledge. Evaluation of 27 models (1B-12B parameters) reveals performance disparities across regions and model scales, indicating systematic gaps in pretraining coverage.
We introduce CARTE 1 (Culturally Anchored Regional-Territorial Evaluation), a multiplechoice benchmark for evaluating the ability of large language models (LLMs) to perform fine-grained reasoning over geographically grounded and regionally differentiated knowledge within France. While prior benchmarks focus on national-level cultural understanding, they largely overlook intra-country variation and the need to distinguish between closely related regional contexts. CARTE addresses this gap by introducing 2,431 questions spanning the 13 metropolitan regions of France and covering 14 thematic domains, including culture, language, demographics, economy, environment, and mobility. We further introduce CARTE-LV, a subset targeting Linguistic Variation across French regions, enabling focused evaluation of language-related differences. We evaluate 27 LLMs ranging from 1B to 12B parameters under few-shot settings. Our experiments reveal performance disparities across regions and model scales, suggesting systematic gaps in pretraining coverage and limited robustness to intra-national variation.