WiCE: Real-World Entailment for Claims in Wikipedia
This addresses the domain shift issue for researchers and practitioners in NLP applications such as fact-checking, but it is incremental as it builds on existing entailment datasets with new data and decomposition methods.
The paper tackles the problem of textual entailment models underperforming in real-world applications like fact-checking due to domain shift, by introducing WiCE, a fine-grained dataset from Wikipedia with claim-level and sub-sentence entailment judgments, and shows that existing models fail on its challenging verification tasks.
Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models' performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.