Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations
This addresses the limitation of LLMs in critically analyzing noisy retrieved contexts for NLP tasks, though it appears incremental as it builds on existing RAG mechanisms.
The paper tackles the problem of improving critical reasoning in retrieval-augmented language models (RAG) by proposing Contrastive-RAG (C-RAG), a framework that generates contrastive explanations to enhance analysis of retrieved documents. The result shows C-RAG improves state-of-the-art RAG models while requiring fewer prompts and demonstrations and being robust to document perturbations.
Retrieval-augmented generation (RAG) has emerged as a critical mechanism in contemporary NLP to support Large Language Models(LLMs) in systematically accessing richer factual context. However, the integration of RAG mechanisms brings its inherent challenges, as LLMs need to deal with potentially noisy contexts. Recent studies have shown that LLMs still struggle to critically analyse RAG-based in-context information, a limitation that may lead to incorrect inferences and hallucinations. In this paper, we investigate how to elicit critical reasoning in RAG via contrastive explanations. In particular, we propose Contrastive-RAG (C-RAG), a framework that (i) retrieves relevant documents given a query, (ii) selects and exemplifies relevant passages, and (iii) generates explanations that explicitly contrast the relevance of the passages to (iv) support the final answer. We show the impact of C-RAG building contrastive reasoning demonstrations from LLMs to instruct smaller models for retrieval-augmented tasks. Extensive experiments demonstrate that C-RAG improves state-of-the-art RAG models while (a) requiring significantly fewer prompts and demonstrations and (b) being robust to perturbations in the retrieved documents.