Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks
This addresses a crucial challenge in machine learning for ensuring model safety and reliability by extending natural language inference to handle sets of statements, though it is incremental as it builds on existing NLI frameworks.
The paper tackles the problem of verifying logical inconsistencies among multiple statements, which traditional pairwise methods often miss, by introducing a set-consistency verification task and a model called SC-Energy that outperforms existing methods, including LLM-based approaches, and releases two new datasets for this task.
Examining logical inconsistencies among multiple statements (such as collections of sentences or question-answer pairs) is a crucial challenge in machine learning, particularly for ensuring the safety and reliability of models. Traditional methods that rely on pairwise comparisons often fail to capture inconsistencies that only emerge when more than two statements are evaluated collectively. To address this gap, we introduce the task of set-consistency verification, an extension of natural language inference (NLI) that assesses the logical coherence of entire sets rather than isolated pairs. Building on this task, we present the Set-Consistency Energy Network (SC-Energy), a novel model that employs a contrastive loss framework to learn the compatibility among a collection of statements. Our approach not only efficiently verifies inconsistencies and pinpoints the specific statements responsible for logical contradictions, but also significantly outperforms existing methods including prompting-based LLM models. Furthermore, we release two new datasets: Set-LConVQA and Set-SNLI for set-consistency verification task.