Multilingual Fine-Grained News Headline Hallucination Detection
This addresses the hallucination issue in automated news headline generation for multilingual applications, offering a more nuanced approach than previous English-focused methods, though it is incremental in extending detection to multiple languages and fine-grained types.
The study tackled the problem of detecting hallucinated news headlines by introducing the first multilingual, fine-grained dataset with over 11,000 pairs in 5 languages, annotated with detailed hallucination types, and proposed novel techniques like language-dependent demonstration selection and coarse-to-fine prompting to improve few-shot detection performance, achieving gains in the example-F1 metric.
The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.