The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking
For fact-checking researchers and practitioners, this work addresses the 'warrant gap' where LLM verdicts lack proper evidential support, offering a practical method to improve reliability.
Fact-checking LLMs often output 'Supports' labels with evidence that does not actually license the claim. The authors propose SIFT, a claim-conditioned re-scoring method, and WSP, an automatic NLI check, which recover up to 27.6 points of accuracy on certain benchmarks and achieve AUC 0.92 and precision 0.98 against human gold evidence.
Fact-checking systems built on LLMs achieve high verdict accuracy on standard benchmarks, yet routinely output Supports labels whose cited evidence does not license the claim. Structured decomposition is the natural way to inspect those warrants, but rigid extraction protocols strip the full-claim context that facets need. We introduce SIFT -- claim-conditioned re-scoring of extracted evidence spans against the full claim -- paired with WSP (Warranted Supports Proportion), an automatic NLI check that the cited warrant entails the claim. We evaluate on FEVER, SciFact, 5PILS, and DP across four open-source backbones. SIFT recovers accuracy on cells where naive decomposition costs up to 27.6 points, while raising WSP above direct prompting; WSP itself calibrates against human gold evidence at AUC 0.92 and precision 0.98.