MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
This work addresses the critical need for reliable factuality evaluation in long-form LLM outputs to guide safe deployment in sensitive domains like biomedicine and law.
The paper tackles the problem of evaluating factual accuracy in long-form outputs from Large Language Models, especially in high-risk domains, by proposing a multi-agent debate framework and new datasets, with experiments showing that larger LLMs generally maintain higher factual consistency.
The widespread adoption of Large Language Models (LLMs) raises critical concerns about the factual accuracy of their outputs, especially in high-risk domains such as biomedicine, law, and education. Existing evaluation methods for short texts often fail on long-form content due to complex reasoning chains, intertwined perspectives, and cumulative information. To address this, we propose a systematic approach integrating large-scale long-form datasets, multi-agent verification mechanisms, and weighted evaluation metrics. We construct LongHalluQA, a Chinese long-form factuality dataset; and develop MAD-Fact, a debate-based multi-agent verification system. We introduce a fact importance hierarchy to capture the varying significance of claims in long-form texts. Experiments on two benchmarks show that larger LLMs generally maintain higher factual consistency, while domestic models excel on Chinese content. Our work provides a structured framework for evaluating and enhancing factual reliability in long-form LLM outputs, guiding their safe deployment in sensitive domains.