Open-Source Reproduction and Explainability Analysis of Corrective Retrieval Augmented Generation
This work provides an open-source alternative for researchers studying RAG robustness, though it is incremental as it replicates an existing method with minor modifications.
The authors reproduced the Corrective Retrieval Augmented Generation (CRAG) system using fully open-source components, achieving comparable performance on PopQA and ARC-Challenge benchmarks, and conducted an explainability analysis showing the retrieval evaluator relies on named entity alignment with domain transfer limitations.
Corrective Retrieval Augmented Generation (CRAG) improves the robustness of RAG systems by evaluating retrieved document quality and triggering corrective actions. However, the original implementation relies on proprietary components including the Google Search API and closed model weights, limiting reproducibility. In this work, we present a fully open-source reproduction of CRAG, replacing proprietary web search with the Wikipedia API and the original LLaMA-2 generator with Phi-3-mini-4k-instruct. We evaluate on PopQA and ARC-Challenge, demonstrating that our open-source pipeline achieves comparable performance to the original system. Furthermore, we contribute the first explainability analysis of CRAG's T5-based retrieval evaluator using SHAP, revealing that the evaluator primarily relies on named entity alignment rather than semantic similarity. Our analysis identifies key failure modes including domain transfer limitations on science questions. All code and results are available at https://github.com/suryayalavarthi/crag-reproduction.