Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets
This addresses the challenge of evaluating and improving model robustness for NLP practitioners, though it is incremental as it builds on existing contrast set concepts.
The paper tackled the problem of expensive human annotation for creating contrast sets to test out-of-distribution performance of pretrained language models by proposing an automatic generation method, which showed that models struggle on these sets and improved performance through data augmentation without harming original data accuracy.
Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e.g., on contrast sets). Building contrast sets often re-quires human-expert annotation, which is expensive and hard to create on a large scale. In this work, we propose a Linguistically-Informed Transformation (LIT) method to automatically generate contrast sets, which enables practitioners to explore linguistic phenomena of interests as well as compose different phenomena. Experimenting with our method on SNLI and MNLI shows that current pretrained language models, although being claimed to contain sufficient linguistic knowledge, struggle on our automatically generated contrast sets. Furthermore, we improve models' performance on the contrast sets by apply-ing LIT to augment the training data, without affecting performance on the original data.