MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verification
This work addresses the need for justifiable and inspectable claim verification in AI, offering a domain-specific advancement in computational argumentation for LLM applications.
The paper tackles the problem of unstructured interactions among multiple LLMs in claim verification, which leads to unjustifiable outputs, by introducing MArgE, a framework that structures evidence as argument trees using Argumentative LLMs, resulting in significant performance improvements over single LLMs and prior multi-LLM methods.
Leveraging outputs from multiple large language models (LLMs) is emerging as a method for harnessing their power across a wide range of tasks while mitigating their capacity for making errors, e.g., hallucinations. However, current approaches to combining insights from multiple LLMs often involve unstructured interactions (e.g., free debate), resulting in model generations that are not faithfully justifiable. In this work, we introduce MArgE, a novel framework to provide formal structure to the evidence from each LLM, in the form of a tree of extracted arguments, for the task of claim verification. We use a variant of Argumentative LLMs (ArgLLMs), i.e. LLMs driven by frameworks and semantics from the field of computational argumentation, to construct structured argument trees for given claims. This process creates an inspectable pathway from the initial arguments to the final claim verification decisions, providing a faithful justification thereof. We show experimentally that MArgE can significantly outperform single LLMs, including three open-source models (4B to 8B parameters), GPT-4o-mini and existing ArgLLMs, as well as prior methods for unstructured multi-LLM debates. We thus demonstrate the advantages of incorporating formal, argumentative reasoning mechanisms when combining multiple LLM outputs.