HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations
This work addresses the need for better evaluation of LLM hallucinations in multilingual settings, though it is incremental as it builds on existing categorization and dataset methods.
The paper tackled the problem of LLMs generating non-factual content by introducing HalluVerse25, a multilingual benchmark dataset for fine-grained hallucinations, and evaluated several LLMs on it to provide insights into their performance in detecting hallucinations across different contexts.
Large Language Models (LLMs) are increasingly used in various contexts, yet remain prone to generating non-factual content, commonly referred to as "hallucinations". The literature categorizes hallucinations into several types, including entity-level, relation-level, and sentence-level hallucinations. However, existing hallucination datasets often fail to capture fine-grained hallucinations in multilingual settings. In this work, we introduce HalluVerse25, a multilingual LLM hallucination dataset that categorizes fine-grained hallucinations in English, Arabic, and Turkish. Our dataset construction pipeline uses an LLM to inject hallucinations into factual biographical sentences, followed by a rigorous human annotation process to ensure data quality. We evaluate several LLMs on HalluVerse25, providing valuable insights into how proprietary models perform in detecting LLM-generated hallucinations across different contexts.