ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence
For researchers and practitioners evaluating language model factuality, this metric addresses the overlooked problem of coexisting supporting and contradicting evidence.
The authors propose ConflictScore, a metric to quantify how well language models handle conflicting evidence in grounding documents, and show it effectively detects overconfident claims and improves truthfulness on TruthfulQA.
Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to capture when both supporting and contradicting evidence coexist. We introduce ConflictScore, a novel metric that quantifies how well a model's response acknowledges conflicting evidence in its grounding documents. Our framework decomposes responses into atomic claims, labels each claim against each grounding document, and then aggregates these labels into two complementary measures: ConflictScore-Count (CS-C), the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), the balance between supporting and contradicting evidence. We develop ConflictBench, a benchmark covering diverse forms of conflicts such as ambiguity, contradiction, and divergent opinions, to systematically evaluate our metric. Experiments show that ConflictScore effectively detects overconfident claims across domains and can serve as a corrective feedback mechanism that improves truthfulness on TruthfulQA.