Finding Generalizable Evidence by Learning to Convince Q&A Models
This work addresses the need for robust and generalizable evidence selection in QA systems, offering incremental improvements over existing methods.
The paper tackles the problem of identifying the strongest supporting evidence for answers in question-answering by training agents to select passage sentences that convince pretrained QA models, resulting in humans correctly answering questions with only about 20% of the full passage and improved model generalization.
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed. We train evidence agents to select the passage sentences that most convince a pretrained QA model of a given answer, if the QA model received those sentences instead of the full passage. Rather than finding evidence that convinces one model alone, we find that agents select evidence that generalizes; agent-chosen evidence increases the plausibility of the supported answer, as judged by other QA models and humans. Given its general nature, this approach improves QA in a robust manner: using agent-selected evidence (i) humans can correctly answer questions with only ~20% of the full passage and (ii) QA models can generalize to longer passages and harder questions.