When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA
This addresses the problem of reliable hallucination detection for multilingual applications, though it is incremental as it builds on existing detection methods with a new dataset.
The authors tackled the challenge of fine-grained, multilingual hallucination detection in LLMs by introducing PsiloQA, a large-scale dataset with span-level annotations across 14 languages, and showed that encoder-based models achieve the strongest performance, with effective cross-lingual generalization and cost-efficiency.
Hallucination detection remains a fundamental challenge for the safe and reliable deployment of large language models (LLMs), especially in applications requiring factual accuracy. Existing hallucination benchmarks often operate at the sequence level and are limited to English, lacking the fine-grained, multilingual supervision needed for a comprehensive evaluation. In this work, we introduce PsiloQA, a large-scale, multilingual dataset annotated with span-level hallucinations across 14 languages. PsiloQA is constructed through an automated three-stage pipeline: generating question-answer pairs from Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse LLMs in a no-context setting, and automatically annotating hallucinated spans using GPT-4o by comparing against golden answers and retrieved context. We evaluate a wide range of hallucination detection methods -- including uncertainty quantification, LLM-based tagging, and fine-tuned encoder models -- and show that encoder-based models achieve the strongest performance across languages. Furthermore, PsiloQA demonstrates effective cross-lingual generalization and supports robust knowledge transfer to other benchmarks, all while being significantly more cost-efficient than human-annotated datasets. Our dataset and results advance the development of scalable, fine-grained hallucination detection in multilingual settings.