Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop
This work addresses the problem of improving adversarial robustness in extractive QA systems for researchers, but it is incremental as it focuses on annotation experiences and analyses rather than novel methods or broad advancements.
The authors tackled the challenge of creating high-quality adversarial data for machine reading comprehension by participating in a workshop task, using a quasi-experimental annotation design with humans and models in the loop. They analyzed factors like successful adversarial attacks, cost, and annotator confidence, providing recommendations for future data collection efforts.
We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.