Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
This work addresses dataset and model limitations in information-seeking QA, which is incremental as it builds on existing benchmarks like Natural Questions and TyDi QA.
The study analyzed why information-seeking question answering is challenging, identifying paragraph selection and answerability prediction as key bottlenecks, and found that with gold paragraphs and abstention knowledge, models outperform humans, but predicting answerability remains difficult.
Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development.