MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
This work addresses the challenge of generalization for researchers and practitioners in natural language processing, though it is incremental as it builds on existing datasets and methods.
The MRQA 2019 shared task tackled the problem of evaluating generalization in reading comprehension systems by adapting 18 datasets into a unified format, with the best system achieving an average F1 score of 72.5 on held-out datasets, a 10.7-point improvement over a BERT baseline.
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.