LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents
This provides a domain-specific benchmark for improving contradiction detection in legal AI systems, though it is incremental as it builds on existing RAG and contradiction detection methods.
The paper tackles the problem of unresolved contradictions in retrieved evidence for legal Retrieval-Augmented Generation (RAG) systems, which cause hallucinations and unsound outputs, by presenting a multi-agent framework that generates synthetic legal documents with six structured contradiction types to enable systematic stress-testing and evaluation.
Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with external sources, but unresolved contradictions in retrieved evidence often lead to hallucinations and legally unsound outputs. Benchmarks currently used for contradiction detection lack domain realism, cover only limited conflict types, and rarely extend beyond single-sentence pairs, making them unsuitable for legal applications. Controlled generation of documents with embedded contradictions is therefore essential: it enables systematic stress-testing of models, ensures coverage of diverse conflict categories, and provides a reliable basis for evaluating contradiction detection and resolution. We present a multi-agent contradiction-aware benchmark framework for the legal domain that generates synthetic legal-style documents, injects six structured contradiction types, and models both self- and pairwise inconsistencies. Automated contradiction mining is combined with human-in-the-loop validation to guarantee plausibility and fidelity. This benchmark offers one of the first structured resources for contradiction-aware evaluation in legal RAG pipelines, supporting more consistent, interpretable, and trustworthy systems.