Synthetic Target Domain Supervision for Open Retrieval QA
This addresses the robustness of open retrieval QA for real-world applications in specialized domains, representing an incremental improvement.
The paper tackled the problem of neural passage retrieval under domain shift, finding that Dense Passage Retriever (DPR) lags behind BM25 in specialized domains like COVID-19, and improved it by fine-tuning with synthetic training examples, giving DPR a sizable advantage over BM25 in out-of-domain settings.
Neural passage retrieval is a new and promising approach in open retrieval question answering. In this work, we stress-test the Dense Passage Retriever (DPR) -- a state-of-the-art (SOTA) open domain neural retrieval model -- on closed and specialized target domains such as COVID-19, and find that it lags behind standard BM25 in this important real-world setting. To make DPR more robust under domain shift, we explore its fine-tuning with synthetic training examples, which we generate from unlabeled target domain text using a text-to-text generator. In our experiments, this noisy but fully automated target domain supervision gives DPR a sizable advantage over BM25 in out-of-domain settings, making it a more viable model in practice. Finally, an ensemble of BM25 and our improved DPR model yields the best results, further pushing the SOTA for open retrieval QA on multiple out-of-domain test sets.