Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals
This addresses robustness issues in sentiment classification for NLP practitioners, though it appears incremental as it builds on existing counterfactual data augmentation approaches.
The paper tackles the problem of language models relying on spurious patterns in training data, which limits out-of-distribution performance, by proposing a method that requires only 1% manually annotated counterfactuals and automatically generates additional ones in vector space. The approach achieves accuracy improvements of +3% compared to adding 100% in-distribution samples and +1.3% compared to alternate counterfactual methods on sentiment classification tasks.
For text classification tasks, finetuned language models perform remarkably well. Yet, they tend to rely on spurious patterns in training data, thus limiting their performance on out-of-distribution (OOD) test data. Among recent models aiming to avoid this spurious pattern problem, adding extra counterfactual samples to the training data has proven to be very effective. Yet, counterfactual data generation is costly since it relies on human annotation. Thus, we propose a novel solution that only requires annotation of a small fraction (e.g., 1%) of the original training data, and uses automatic generation of extra counterfactuals in an encoding vector space. We demonstrate the effectiveness of our approach in sentiment classification, using IMDb data for training and other sets for OOD tests (i.e., Amazon, SemEval and Yelp). We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals: +3% compared to adding +100% in-distribution training samples, +1.3% compared to alternate counterfactual approaches.