Repartitioning of the ComplexWebQuestions Dataset
This work addresses a critical data integrity problem for researchers using the ComplexWebQuestions dataset, ensuring more reliable benchmarking in question-answering tasks.
The authors identified data leakage from the training to test set in the ComplexWebQuestions dataset, which led to inflated performance metrics, and they created a new partitioning to eliminate this issue, resulting in improved state-of-the-art performance when training a reading comprehension model on the corrected data.
Recently, Talmor and Berant (2018) introduced ComplexWebQuestions - a dataset focused on answering complex questions by decomposing them into a sequence of simpler questions and extracting the answer from retrieved web snippets. In their work the authors used a pre-trained reading comprehension (RC) model (Salant and Berant, 2018) to extract the answer from the web snippets. In this short note we show that training a RC model directly on the training data of ComplexWebQuestions reveals a leakage from the training set to the test set that allows to obtain unreasonably high performance. As a solution, we construct a new partitioning of ComplexWebQuestions that does not suffer from this leakage and publicly release it. We also perform an empirical evaluation on these two datasets and show that training a RC model on the training data substantially improves state-of-the-art performance.