Fine-grained Claim-level RAG Benchmark for Law
This work provides a more granular evaluation framework for legal RAG systems, addressing the lack of fine-grained, multilingual benchmarks for both expert and non-expert users.
The paper introduces ClaimRAG-LAW, a bilingual (French/English) benchmark for fine-grained evaluation of legal RAG systems, targeting both expert and non-expert queries. The evaluation reveals limitations in retrieval and generation performance, with claim-level analysis showing that even state-of-the-art systems hallucinate at varying rates.
The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses. In high-stake domains such as law, retrieval-augmented generation (RAG) is commonly used to mitigate hallucinations in generated responses. Nonetheless, prior work shows that RAG systems, whether general-purpose or legal-specific, still hallucinate at varying rates, making fine-grained evaluation essential. Despite the need, existing evaluation frameworks for legal RAG systems lack the granularity required to provide detailed analysis of retrieval and generation performance separately. Moreover, current benchmarks are largely English-only and centered on legal expert queries, overlooking non-expert needs. We introduce ClaimRAG-LAW, a comprehensive dataset for legal RAG that supports French and English, targets both experts and non-experts, and includes diverse question types reflecting realistic scenarios. We further apply a fine-grained evaluation framework of state-of-the-art legal RAG systems, revealing limitations in retrieval, generation, and claim-level analysis in the legal domain.