Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
This addresses the limitation in standard evaluations for semantics in NLP, providing a method to stress-test models, though it is incremental in improving evaluation techniques.
The paper tackled the problem of evaluating deep learning models for natural language inference by generating data sets with precisely characterized semantic complexity, showing that common architectures fail to encode crucial information, while only a model with forced lexical alignments avoids this loss.
Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity. To overcome this limitation, we present a method for generating data sets of multiply-quantified natural language inference (NLI) examples in which semantic complexity can be precisely characterized, and we use this method to show that a variety of common architectures for NLI inevitably fail to encode crucial information; only a model with forced lexical alignments avoids this damaging information loss.