Comparing Hallucination Detection Metrics for Multilingual Generation
This work addresses the problem of evaluating hallucination detection for non-English languages, which is incremental as it extends existing methods to a multilingual context.
This paper assessed the effectiveness of various hallucination detection metrics for multilingual generated text, finding that NLI-based metrics performed well by correlating with human judgments in many settings, while lexical metrics were ineffective and NLI metrics had limitations like poor detection of single-fact hallucinations and failure in lower-resource languages.
While many hallucination detection techniques have been evaluated on English text, their effectiveness in multilingual contexts remains unknown. This paper assesses how well various factual hallucination detection metrics (lexical metrics like ROUGE and Named Entity Overlap, and Natural Language Inference (NLI)-based metrics) identify hallucinations in generated biographical summaries across languages. We compare how well automatic metrics correlate to each other and whether they agree with human judgments of factuality. Our analysis reveals that while the lexical metrics are ineffective, NLI-based metrics perform well, correlating with human annotations in many settings and often outperforming supervised models. However, NLI metrics are still limited, as they do not detect single-fact hallucinations well and fail for lower-resource languages. Therefore, our findings highlight the gaps in exisiting hallucination detection methods for non-English languages and motivate future research to develop more robust multilingual detection methods for LLM hallucinations.