MuSiQue: Multihop Questions via Single-hop Question Composition
This addresses the need for better benchmarks in NLP to develop models capable of genuine multihop reasoning, though it is incremental as it builds on existing dataset creation methods.
The authors tackled the problem of creating a multihop question answering dataset that requires genuine reasoning by introducing a bottom-up approach to compose single-hop questions, resulting in MuSiQue-Ans with 25K questions that shows a 3x increase in human-machine gap and a 30-point F1 drop for single-hop models.
Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, \emph{requires} proper multihop reasoning? To this end, we introduce a bottom-up approach that systematically selects composable pairs of single-hop questions that are connected, i.e., where one reasoning step critically relies on information from another. This bottom-up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting $k$-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2-4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3x increase in human-machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30 point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.