Better Retrieval May Not Lead to Better Question Answering
This challenges a common assumption in open-domain QA for researchers, indicating an incremental insight into system limitations.
The paper tackles the problem of improving open-domain question answering by enhancing retrieval quality, finding that for the StrategyQA dataset, better retrieval does not lead to better performance, with no concrete numbers provided.
Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC). A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. In this work we show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective -- improving the quality of the retrieved context hardly improves the system's performance. We further analyze the system's behavior to identify potential reasons.