Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models
This work addresses the problem of multilingual knowledge gaps in AI models for researchers and developers, but it is incremental as it benchmarks existing models without proposing new methods.
The study assessed foundation models' ability to recall encyclopedic knowledge across languages, finding that Meta's LLaMA performed best but had significant limitations in non-English languages and with fact subjects like location and gender.
In this work, we assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with counterfactuals, 2) evaluate 5 models in a multilingual test, and 3) benchmark a diverse set of 24 models in an English-only test. Meta's LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA's errors reveals significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. Overall, our findings suggest that today's foundation models are far from polyglots.