LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI
This work addresses the need for more comprehensive testing of reasoning abilities in NLI systems, which is crucial for advancing natural language understanding, though it is incremental as it builds on existing frameworks like CheckList.
The authors tackled the problem of testing diverse logical reasoning capabilities in Natural Language Inference (NLI) by proposing an extensible framework with a semi-synthetic test bench of 363 templates and 363k examples, which they found to be hard for state-of-the-art systems, with some capabilities being particularly challenging.
Natural Language Inference (NLI) is considered a representative task to test natural language understanding (NLU). In this work, we propose an extensible framework to collectively yet categorically test diverse Logical reasoning capabilities required for NLI (and, by extension, NLU). Motivated by behavioral testing, we create a semi-synthetic large test bench (363 templates, 363k examples) and an associated framework that offers the following utilities: 1) individually test and analyze reasoning capabilities along 17 reasoning dimensions (including pragmatic reasoning); 2) design experiments to study cross-capability information content (leave one out or bring one in); and 3) the synthetic nature enables us to control for artifacts and biases. We extend a publicly available framework of automated test case instantiation from free-form natural language templates (CheckList) and a well-defined taxonomy of capabilities to cover a wide range of increasingly harder test cases while varying the complexity of natural language. Through our analysis of state-of-the-art NLI systems, we observe that our benchmark is indeed hard (and non-trivial even with training on additional resources). Some capabilities stand out as harder. Further, fine-grained analysis and fine-tuning experiments reveal more insights about these capabilities and the models -- supporting and extending previous observations; thus showing the utility of the proposed testbench.