AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
It addresses the underexplored issue of hallucination evaluation for Arabic LLMs, which is important for users in Arabic-speaking regions, but is incremental as it applies existing evaluation concepts to a new language context.
This paper tackles the problem of evaluating hallucination in Arabic and multilingual large language models (LLMs) on generative question answering and summarization tasks, finding that factual hallucinations are more common than faithfulness errors and that the Arabic pre-trained model Allam shows lower hallucination rates than multilingual models and performs comparably to reasoning-based models.
Recently, extensive research on the hallucination of the large language models (LLMs) has mainly focused on the English language. Despite the growing number of multilingual and Arabic-specific LLMs, evaluating LLMs' hallucination in the Arabic context remains relatively underexplored. The knowledge gap is particularly pressing given Arabic's widespread use across many regions and its importance in global communication and media. This paper presents the first comprehensive hallucination evaluation of Arabic and multilingual LLMs on two critical Arabic natural language generation tasks: generative question answering (GQA) and summarization. This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models. To assess the factual consistency and faithfulness of LLMs' outputs, we developed a fine-grained hallucination evaluation framework consisting of 12 fine-grained hallucination indicators that represent the varying characteristics of each task. The results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks. Notably, the Arabic pre-trained model Allam consistently demonstrates lower hallucination rates than multilingual models and a comparative performance with reasoning-based models. The code is available at: https://github.com/aishaalansari57/AraHalluEval