Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering
This work addresses evidence retrieval for explainable QA, offering a faster and more effective hybrid method, though it is incremental as it builds on existing IR and neural techniques.
The paper tackles the problem of evidence retrieval for question answering by showing that neural methods like USE-QA sometimes underperform traditional IR methods like BM25, and introduces a hybrid approach that combines both, achieving better performance on three QA datasets and up to 5 times faster runtime than USE-QA.
Evidence retrieval is a key component of explainable question answering (QA). We argue that, despite recent progress, transformer network-based approaches such as universal sentence encoder (USE-QA) do not always outperform traditional information retrieval (IR) methods such as BM25 for evidence retrieval for QA. We introduce a lexical probing task that validates this observation: we demonstrate that neural IR methods have the capacity to capture lexical differences between questions and answers, but miss obvious lexical overlap signal. Learning from this probing analysis, we introduce a hybrid approach for evidence retrieval that combines the advantages of both IR directions. Our approach uses a routing classifier that learns when to direct incoming questions to BM25 vs. USE-QA for evidence retrieval using very simple statistics, which can be efficiently extracted from the top candidate evidence sentences produced by a BM25 model. We demonstrate that this hybrid evidence retrieval generally performs better than either individual retrieval strategy on three QA datasets: OpenBookQA, ReQA SQuAD, and ReQA NQ. Furthermore, we show that the proposed routing strategy is considerably faster than neural methods, with a runtime that is up to 5 times faster than USE-QA.