ANLIzing the Adversarial Natural Language Inference Dataset
This work provides incremental insights for NLP researchers by enabling more detailed evaluation of models on adversarial reasoning tasks.
The authors performed an error analysis of the Adversarial NLI dataset by developing a fine-grained annotation scheme to categorize inference types, identifying which types are most common, challenging, and where models perform best.
We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over multiple rounds. We propose a fine-grained annotation scheme of the different aspects of inference that are responsible for the gold classification labels, and use it to hand-code all three of the ANLI development sets. We use these annotations to answer a variety of interesting questions: which inference types are most common, which models have the highest performance on each reasoning type, and which types are the most challenging for state of-the-art models? We hope that our annotations will enable more fine-grained evaluation of models trained on ANLI, provide us with a deeper understanding of where models fail and succeed, and help us determine how to train better models in future.