Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
This provides a new benchmark for NLP researchers to better understand and improve negation capabilities in language models, though it is incremental as it builds on existing NLI frameworks.
The authors tackled the problem of poor negation handling in language models by introducing the NaN-NLI test suite for sub-clausal negation, showing it is more challenging than existing benchmarks and enabling fine-grained analysis of model performance.
Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods, with the aim of understanding sub-clausal negation. The test suite contains premise--hypothesis pairs where the premise contains sub-clausal negation and the hypothesis is constructed by making minimal modifications to the premise in order to reflect different possible interpretations. Aside from adopting standard NLI labels, our test suite is systematically constructed under a rigorous linguistic framework. It includes annotation of negation types and constructions grounded in linguistic theory, as well as the operations used to construct hypotheses. This facilitates fine-grained analysis of model performance. We conduct experiments using pre-trained language models to demonstrate that our test suite is more challenging than existing benchmarks focused on negation, and show how our annotation supports a deeper understanding of the current NLI capabilities in terms of negation and quantification.