Augmenting NLP data to counter Annotation Artifacts for NLI Tasks
This addresses a critical issue in NLP for researchers and practitioners by mitigating biases in NLI datasets, though it is incremental as it builds on known problems with annotation artifacts.
The paper tackles the problem of annotation artifacts in Natural Language Inference (NLI) tasks, where models rely on dataset biases rather than solving the underlying task, and proposes a data augmentation technique to fix this bias, showing its effectiveness through measurement.
In this paper, we explore Annotation Artifacts - the phenomena wherein large pre-trained NLP models achieve high performance on benchmark datasets but do not actually "solve" the underlying task and instead rely on some dataset artifacts (same across train, validation, and test sets) to figure out the right answer. We explore this phenomenon on the well-known Natural Language Inference task by first using contrast and adversarial examples to understand limitations to the model's performance and show one of the biases arising from annotation artifacts (the way training data was constructed by the annotators). We then propose a data augmentation technique to fix this bias and measure its effectiveness.