Robust Machine Comprehension Models via Adversarial Training
This addresses robustness issues in question-answering models for NLP applications, but it is incremental as it builds on prior adversarial training methods.
The paper tackled the problem of machine comprehension models lacking robustness to adversarial examples, showing that existing models suffer over 50% F1 score drops, and proposed a method that increased robustness by 36.5% in F1 score under adversarial evaluation while maintaining performance on the standard task.
It is shown that many published models for the Stanford Question Answering Dataset (Rajpurkar et al., 2016) lack robustness, suffering an over 50% decrease in F1 score during adversarial evaluation based on the AddSent (Jia and Liang, 2017) algorithm. It has also been shown that retraining models on data generated by AddSent has limited effect on their robustness. We propose a novel alternative adversary-generation algorithm, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions. Further, in order to improve robustness to AddSent's semantic perturbations (e.g., antonyms), we jointly improve the model's semantic-relationship learning capabilities in addition to our AddSentDiverse-based adversarial training data augmentation. With these additions, we show that we can make a state-of-the-art model significantly more robust, achieving a 36.5% increase in F1 score under many different types of adversarial evaluation while maintaining performance on the regular SQuAD task.