Foundations of Global Consistency Checking with Noisy LLM Oracles
This addresses the challenge of ensuring coherence in tasks like fact-checking and knowledge base construction with LLMs, offering a scalable solution for linguistic consistency verification.
The paper tackles the problem of verifying global consistency among collections of natural-language facts using noisy LLM oracles, showing that worst-case verification requires exponential queries, and proposes an adaptive divide-and-conquer algorithm with low-degree polynomial query complexity that efficiently detects and localizes inconsistencies in experiments.
Ensuring that collections of natural-language facts are globally consistent is essential for tasks such as fact-checking, summarization, and knowledge base construction. While Large Language Models (LLMs) can assess the consistency of small subsets of facts, their judgments are noisy, and pairwise checks are insufficient to guarantee global coherence. We formalize this problem and show that verifying global consistency requires exponentially many oracle queries in the worst case. To make the task practical, we propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets. Our approach has low-degree polynomial query complexity. Experiments with both synthetic and real LLM oracles show that our method efficiently detects and localizes inconsistencies, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.