AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
This addresses the challenge of trustworthy LLMs by reducing the cost of hallucination dataset creation, though it is incremental as it builds on existing fact-checking datasets.
The paper tackles the problem of laborious and expensive manual annotation for hallucination detection in large language models by proposing AutoHall, a method for automatically generating model-specific hallucination datasets from existing fact-checking data, which reveals variations in hallucination proportions and types among models and enables a zero-resource detection method achieving superior performance.
Large language models (LLMs) have gained broad applications across various domains but still struggle with hallucinations. Currently, hallucinations occur frequently in the generation of factual content and pose a great challenge to trustworthy LLMs. However, hallucination detection is hindered by the laborious and expensive manual annotation of hallucinatory content. Meanwhile, as different LLMs exhibit distinct types and rates of hallucination, the collection of hallucination datasets is inherently model-specific, which also increases the cost. To address this issue, this paper proposes a method called $\textbf{AutoHall}$ for $\underline{Auto}$matically constructing model-specific $\underline{Hall}$ucination datasets based on existing fact-checking datasets. The empirical results reveal variations in hallucination proportions and types among different models. Moreover, we introduce a zero-resource and black-box hallucination detection method based on self-contradiction to recognize the hallucination in our constructed dataset, achieving superior detection performance compared to baselines. Further analysis on our dataset provides insight into factors that may contribute to LLM hallucinations. Our codes and datasets are publicly available at https://github.com/zouyingcao/AutoHall.