When is dataset cartography ineffective? Using training dynamics does not improve robustness against Adversarial SQuAD
This work addresses adversarial robustness for SQuAD-style QA tasks, showing incremental results with limited practical benefit.
The study investigated whether dataset cartography improves adversarial robustness in extractive question answering on SQuAD, finding that training on subsets based on training dynamics did not enhance generalization to validation or adversarial sets, with only minor gains on one dataset.
In this paper, I investigate the effectiveness of dataset cartography for extractive question answering on the SQuAD dataset. I begin by analyzing annotation artifacts in SQuAD and evaluate the impact of two adversarial datasets, AddSent and AddOneSent, on an ELECTRA-small model. Using training dynamics, I partition SQuAD into easy-to-learn, ambiguous, and hard-to-learn subsets. I then compare the performance of models trained on these subsets to those trained on randomly selected samples of equal size. Results show that training on cartography-based subsets does not improve generalization to the SQuAD validation set or the AddSent adversarial set. While the hard-to-learn subset yields a slightly higher F1 score on the AddOneSent dataset, the overall gains are limited. These findings suggest that dataset cartography provides little benefit for adversarial robustness in SQuAD-style QA tasks. I conclude by comparing these results to prior findings on SNLI and discuss possible reasons for the observed differences.