Exploring and Improving Robustness of Multi Task Deep Neural Networks via Domain Agnostic Defenses
This addresses robustness issues in multi-task learning for NLP, offering a defense method, but it is incremental as it builds on prior work showing MT-DNNs are more robust than single-task models.
The paper tackles the vulnerability of Multi-Task Deep Neural Networks (MT-DNNs) to adversarial attacks in Natural Language Understanding tasks, showing that accuracy drops by up to 42.05% after attacks, and proposes a domain-agnostic defense that restores accuracy by up to 36.75%.
In this paper, we explore the robustness of the Multi-Task Deep Neural Networks (MT-DNN) against non-targeted adversarial attacks across Natural Language Understanding (NLU) tasks as well as some possible ways to defend against them. Liu et al., have shown that the Multi-Task Deep Neural Network, due to the regularization effect produced when training as a result of its cross task data, is more robust than a vanilla BERT model trained only on one task (1.1%-1.5% absolute difference). We further show that although the MT-DNN has generalized better, making it easily transferable across domains and tasks, it can still be compromised as after only 2 attacks (1-character and 2-character) the accuracy drops by 42.05% and 32.24% for the SNLI and SciTail tasks. Finally, we propose a domain agnostic defense which restores the model's accuracy (36.75% and 25.94% respectively) as opposed to a general-purpose defense or an off-the-shelf spell checker.