A multilingual hallucination benchmark: MultiWikiQHalluA
For NLP researchers and practitioners working on multilingual models, this work provides a benchmark and analysis of hallucination rates across languages, highlighting disparities for lower-resource languages.
The authors created synthetic hallucination datasets for 306 languages using MultiWikiQA and LettuceDetect, then trained token-level classifiers for 30 European languages. Evaluating five models on English, Danish, German, and Icelandic, they found hallucination rates up to 60% for Qwen3-0.6B (peaking in Icelandic), with larger models performing better and higher rates for lower-resource languages.
Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60\% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.