Synthetic Question Value Estimation for Domain Adaptation of Question Answering
This addresses domain adaptation for question answering models by reducing reliance on human annotations, though it is incremental as it builds on existing synthetic question generation methods.
The paper tackles the problem of noisy synthetic questions in domain adaptation for question answering by introducing a question value estimator (QVE) that directly estimates their usefulness for improving target-domain QA performance, achieving better results than existing techniques and comparable performance to fully-supervised baselines with only around 15% of human annotations.
Synthesizing QA pairs with a question generator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models. Since synthetic questions are often noisy in practice, existing work adapts scores from a pretrained QA (or QG) model as criteria to select high-quality questions. However, these scores do not directly serve the ultimate goal of improving QA performance on the target domain. In this paper, we introduce a novel idea of training a question value estimator (QVE) that directly estimates the usefulness of synthetic questions for improving the target-domain QA performance. By conducting comprehensive experiments, we show that the synthetic questions selected by QVE can help achieve better target-domain QA performance, in comparison with existing techniques. We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable performance to the fully-supervised baselines.