Unleashing Potential of Evidence in Knowledge-Intensive Dialogue Generation
This work addresses the challenge of improving factual accuracy in dialogue systems for applications requiring reliable knowledge integration, though it is incremental as it builds on existing evidence retrieval methods.
The paper tackles the problem of irrelevant content and hallucinations in knowledge-intensive dialogue generation by proposing a framework that uses LLMs to generate evidence labels and an evidence-focused attention mechanism, resulting in improvements of +3 to +5 points in coherence and factual consistency on the MultiDoc2Dial dataset.
Incorporating external knowledge into dialogue generation (KIDG) is crucial for improving the correctness of response, where evidence fragments serve as knowledgeable snippets supporting the factual dialogue replies. However, introducing irrelevant content often adversely impacts reply quality and easily leads to hallucinated responses. Prior work on evidence retrieval and integration in dialogue systems falls short of fully leveraging existing evidence since the model fails to locate useful fragments accurately and overlooks hidden evidence labels within the KIDG dataset. To fully Unleash the potential of evidence, we propose a framework to effectively incorporate Evidence in knowledge-Intensive Dialogue Generation (u-EIDG). Specifically, we introduce an automatic evidence generation framework that harnesses the power of Large Language Models (LLMs) to mine reliable evidence veracity labels from unlabeled data. By utilizing these evidence labels, we train a reliable evidence indicator to effectively identify relevant evidence from retrieved passages. Furthermore, we propose an evidence-augmented generator with an evidence-focused attention mechanism, which allows the model to concentrate on evidenced segments. Experimental results on MultiDoc2Dial demonstrate the efficacy of evidential label augmentation and refined attention mechanisms in improving model performance. Further analysis confirms that the proposed method outperforms other baselines (+3~+5 points) regarding coherence and factual consistency.