On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
This work highlights critical dataset quality issues affecting conversational AI reliability, with implications for researchers and developers in the field.
The study investigated whether hallucinations in knowledge-grounded conversational models stem from training data or models, finding that standard benchmarks contain over 60% hallucinated responses, causing models to amplify these errors.
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of >60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.