Improving QA Generalization by Concurrent Modeling of Multiple Biases
This addresses the issue of poor generalization in question answering for NLP researchers and practitioners, though it is incremental as it builds on existing debiasing methods.
The paper tackles the problem of NLP models exploiting dataset-specific biases, which limits generalization, by proposing a framework that weights examples based on multiple biases to reduce reliance on biased data, showing effectiveness in both single- and multi-domain settings compared to state-of-the-art debiasing methods.
Existing NLP datasets contain various biases that models can easily exploit to achieve high performances on the corresponding evaluation sets. However, focusing on dataset-specific biases limits their ability to learn more generalizable knowledge about the task from more general data patterns. In this paper, we investigate the impact of debiasing methods for improving generalization and propose a general framework for improving the performance on both in-domain and out-of-domain datasets by concurrent modeling of multiple biases in the training data. Our framework weights each example based on the biases it contains and the strength of those biases in the training data. It then uses these weights in the training objective so that the model relies less on examples with high bias weights. We extensively evaluate our framework on extractive question answering with training data from various domains with multiple biases of different strengths. We perform the evaluations in two different settings, in which the model is trained on a single domain or multiple domains simultaneously, and show its effectiveness in both settings compared to state-of-the-art debiasing methods.