Improving QA Model Performance with Cartographic Inoculation
This work addresses the issue of poor generalization in QA models for real-world applications, though it is incremental as it builds on existing methods for mitigating dataset artifacts.
The paper tackled the problem of QA models exploiting dataset artifacts, which harms generalization, by proposing cartographic inoculation, a method that fine-tunes models on an optimized subset of adversarial challenge data, resulting in significant performance improvements on the challenge dataset with minimal loss of generalizability.
QA models are faced with complex and open-ended contextual reasoning problems, but can often learn well-performing solution heuristics by exploiting dataset-specific patterns in their training data. These patterns, or "dataset artifacts", reduce the model's ability to generalize to real-world QA problems. Utilizing an ElectraSmallDiscriminator model trained for QA, we analyze the impacts and incidence of dataset artifacts using an adversarial challenge set designed to confuse models reliant on artifacts for prediction. Extending existing work on methods for mitigating artifact impacts, we propose cartographic inoculation, a novel method that fine-tunes models on an optimized subset of the challenge data to reduce model reliance on dataset artifacts. We show that by selectively fine-tuning a model on ambiguous adversarial examples from a challenge set, significant performance improvements can be made on the full challenge dataset with minimal loss of model generalizability to other challenging environments and QA datasets.