Long-Form Information Alignment Evaluation Beyond Atomic Facts
This addresses a critical vulnerability in NLG evaluation for trustworthy LLM deployment, though it is an incremental improvement over fine-grained methods.
The paper tackles the problem of evaluating information alignment in long-form text, where current methods fail to detect deceptive narratives created by combining truthful statements. They introduce MontageLie, a benchmark showing existing evaluators' vulnerability (AUC-ROC <65%), and propose DoveScore, which improves performance by over 8% by verifying factual accuracy and event-order consistency.
Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.