Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
This work addresses the efficiency-accuracy trade-off in neural retrieval for question answering, offering a method to enhance late fusion models with early fusion insights, which is incremental but provides measurable improvements.
The paper tackles the problem of improving neural retrieval models for question answering by using a supervised data augmentation method, where an accurate early fusion model annotates additional passages to train a late fusion retrieval model, resulting in significant performance gains on Precision at N and Mean Reciprocal Rank metrics.
Neural models that independently project questions and answers into a shared embedding space allow for efficient continuous space retrieval from large corpora. Independently computing embeddings for questions and answers results in late fusion of information related to matching questions to their answers. While critical for efficient retrieval, late fusion underperforms models that make use of early fusion (e.g., a BERT based classifier with cross-attention between question-answer pairs). We present a supervised data mining method using an accurate early fusion model to improve the training of an efficient late fusion retrieval model. We first train an accurate classification model with cross-attention between questions and answers. The accurate cross-attention model is then used to annotate additional passages in order to generate weighted training examples for a neural retrieval model. The resulting retrieval model with additional data significantly outperforms retrieval models directly trained with gold annotations on Precision at $N$ (P@N) and Mean Reciprocal Rank (MRR).