Non-entailed subsequences as a challenge for natural language inference
This addresses a challenge for NLI systems by exposing a heuristic that may hinder deep language understanding, though it is incremental as it builds on prior studies of model limitations.
The paper tackles the problem of neural network models in natural language inference (NLI) potentially relying on fallible heuristics, specifically testing if they assume a sentence entails all its subsequences, and finds strong evidence that competitive models do rely on this heuristic.
Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis. However, recent studies suggest that these models may rely on fallible heuristics rather than deep language understanding. We introduce a challenge set to test whether NLI systems adopt one such heuristic: assuming that a sentence entails all of its subsequences, such as assuming that "Alice believes Mary is lying" entails "Alice believes Mary." We evaluate several competitive NLI models on this challenge set and find strong evidence that they do rely on the subsequence heuristic.