Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems
This addresses the issue of factual inaccuracies in customer service and similar knowledge-intensive dialog systems, but it is incremental as it builds on existing RAG and agent approaches.
The paper tackles the problem of large language models (LLMs) making errors in knowledge-intensive dialog systems by proposing knowledge augmented finetuning (KAFT), which finetunes LLMs with domain-specific data and external knowledge, and shows it substantially surpasses prompting in factual accuracy for RAG and agent-based systems on the MobileCS2 dataset.
Large language models (LLMs) have recently been applied to dialog systems. Despite making progress, LLMs are prone to errors in knowledge-intensive scenarios. Recently, approaches based on retrieval augmented generation (RAG) and agent have emerged to improve the factual accuracy by enhancing the LLMs with knowledge retrieved from external knowledge bases (KBs). This is mostly implemented by prompting the LLMs with instructions, examples and the retrieved knowledge. However, LLMs may have difficulty using the retrieved knowledge effectively for response generation, because they are not well trained to do such generation for specific domains. To mitigate this problem, we propose to finetune the LLMs in the RAG-based and agent-based systems with domain-specific data, together with domain-specific external knowledge, which is called knowledge augmented finetuning (KAFT). We base our study on the MobileCS2 dataset, a real-life customer service dialog dataset that features intensive knowledge interactions, to systematically compare the prompting and KAFT techniques in the RAG-based and agent-based systems. Experiment results show that KAFT substantially surpasses prompting in both RAG and agent systems, particularly in terms of factual accuracy. To the best of our knowledge, this paper represents the first solid empirical work to investigate the KAFT idea.