A corpus of precise natural textual entailment problems
This provides a new benchmark for testing precise natural-language inference systems, though it is incremental as it builds on existing datasets.
The authors tackled the problem of evaluating natural language inference systems by creating a corpus of 150 precise textual entailment problems based on real-world texts, with missing hypotheses identified by experts.
In this paper, we present a new corpus of entailment problems. This corpus combines the following characteristics: 1. it is precise (does not leave out implicit hypotheses) 2. it is based on "real-world" texts (i.e. most of the premises were written for purposes other than testing textual entailment). 3. its size is 150. The corpus was constructed by taking problems from the Real Text Entailment and discovering missing hypotheses using a crowd of experts. We believe that this corpus constitutes a first step towards wide-coverage testing of precise natural-language inference systems.