Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference
This work addresses a foundational issue in interpreting model performance on NLI tasks, which is crucial for researchers in natural language understanding, but it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of understanding the logical properties of Natural Language Inference (NLI) by formulating three readings of the NLI label set and analyzing their meta-inferential properties, using the SNLI dataset to evaluate model consistency and derive insights into encoded logical relations.
Natural Language Inference (NLI) has been an important task for evaluating language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to interpreting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail. Focusing on the SNLI dataset, we exploit (1) NLI items with shared premises and (2) items generated by LLMs to evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.