ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges
This work addresses the problem of costly and difficult evaluation for computational argumentation systems, particularly for researchers and developers working on retrieval-augmented generation tasks, though it appears incremental as it builds on existing LLM judge approaches.
The paper tackles the challenge of evaluating Retrieval-Augmented Argumentation (RAArg) by proposing automated evaluation methods using multiple fine-grained LLM judges, which provide better and more interpretable assessments than traditional metrics and human crowdsourcing. To validate these techniques, they introduce ConQRet, a new benchmark featuring long, complex human-authored arguments on debated topics grounded in real-world websites, enabling exhaustive evaluation across retrieval effectiveness, argument quality, and groundedness.
Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today's polarized environment. Sophisticated LLM capabilities offer the potential to provide nuanced, evidence-based answers to such questions through Retrieval-Augmented Argumentation (RAArg), leveraging real-world evidence for high-quality, grounded arguments. However, evaluating RAArg remains challenging, as human evaluation is costly and difficult for complex, lengthy answers on complicated topics. At the same time, re-using existing argumentation datasets is no longer sufficient, as they lack long, complex arguments and realistic evidence from potentially misleading sources, limiting holistic evaluation of retrieval effectiveness and argument quality. To address these gaps, we investigate automated evaluation methods using multiple fine-grained LLM judges, providing better and more interpretable assessments than traditional single-score metrics and even previously reported human crowdsourcing. To validate the proposed techniques, we introduce ConQRet, a new benchmark featuring long and complex human-authored arguments on debated topics, grounded in real-world websites, allowing an exhaustive evaluation across retrieval effectiveness, argument quality, and groundedness. We validate our LLM Judges on a prior dataset and the new ConQRet benchmark. Our proposed LLM Judges and the ConQRet benchmark can enable rapid progress in computational argumentation and can be naturally extended to other complex retrieval-augmented generation tasks.