Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents
Provides a domain-specific benchmark for evaluating RAG systems on defense documents, addressing pretraining overlap and weak attribution issues in open-domain benchmarks.
DoRA introduces a synthetic benchmark for RAG-based QA on defense documents, covering five question types with 6.5K instances. A model fine-tuned on DoRA (DoRA SFT) achieves up to 26% improvement in QA success and reduces hallucination by 47% in RAG faithfulness scores compared to the base model.
Open-domain RAG benchmarks over public corpora can overestimate deployment performance due to pretraining overlap and weak attribution requirements. We present DoRA (Domain-oriented RAG Assessment), a domain-grounded benchmark built from defense documents that pairs synthetic, intent-conditioned QA (question answering) with auditable evidence passages for attribution. DoRA covers five question types (find, explain, summarize, generate, provide) and contains 6.5K curated instances. In end-to-end evaluation with a fixed dense retriever, general-purpose Language Models (LMs) perform similarly, while a model trained on DoRA (DoRA SFT) yields large gains over the base model (Llama3.1-8B-Instruct): up to 26% improvement in QA task success, while reducing the hallucination rate by 47% in RAG faithfulness scores, supporting contamination-aware regression testing under domain shift.