Improving LLM's Attachment to External Knowledge In Dialogue Generation Tasks Through Entity Anonymization
This addresses the issue of LLMs underutilizing external knowledge in dialogue tasks, which is incremental as it builds on existing KG-DG methods.
The paper tackled the problem of large language models (LLMs) detaching from external knowledge in knowledge graph-based dialogue generation, and the result was that their entity anonymization technique improved attachment on the OpenDialKG dataset.
Knowledge graph-based dialogue generation (KG-DG) is a challenging task requiring models to effectively incorporate external knowledge into conversational responses. While large language models (LLMs) have achieved impressive results across various NLP tasks, their ability to utilize external knowledge in KG-DG remains under-explored. We observe that LLMs often rely on internal knowledge, leading to detachment from provided knowledge graphs, even when they are given a flawlessly retrieved knowledge graph. First, we introduce LLM-KAT, an evaluation procedure for measuring knowledge attachment in generated responses. Second, we propose a simple yet effective entity anonymization technique to encourage LLMs to better leverage external knowledge. Experiments on the OpenDialKG dataset demonstrate that our approach improves LLMs' attachment on external knowledge.