An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought
This work addresses the need for better performance in generative dialogue models for users handling complex reasoning tasks, but it is incremental as it builds on existing methods like RAG and CoT.
The paper tackled improving generative dialogue models for complex problems by evaluating the RAFT method, which combines chain-of-thought, supervised fine-tuning, and retrieval augmented generation, showing significant enhancements in information extraction and logical reasoning across multiple datasets including long-form QA and Chinese tasks.
Since the launch of ChatGPT at the end of 2022, generative dialogue models represented by ChatGPT have quickly become essential tools in daily life. As user expectations increase, enhancing the capability of generative dialogue models to solve complex problems has become a focal point of current research. This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. We evaluated the RAFT method across multiple datasets and analysed its performance in various reasoning tasks, including long-form QA and short-form QA tasks, tasks in both Chinese and English, and supportive and comparison reasoning tasks. Notably, it addresses the gaps in previous research regarding long-form QA tasks and Chinese datasets. Moreover, we also evaluate the benefit of the chain-of-thought (CoT) in the RAFT method. This work offers valuable insights for studies focused on enhancing the performance of generative dialogue models.