Benchmarking Popular Classification Models' Robustness to Random and Targeted Corruptions
This work addresses the robustness gap for text classification models in real-world applications, but it is incremental as it extends existing datasets with new corruptions.
The authors tackled the problem of text classification models performing poorly on real-world data by creating a model-agnostic benchmark dataset with natural corruptions like spelling errors and synonyms, and found that targeted corruptions expose model vulnerabilities better than random ones in most cases.
Text classification models, especially neural networks based models, have reached very high accuracy on many popular benchmark datasets. Yet, such models when deployed in real world applications, tend to perform badly. The primary reason is that these models are not tested against sufficient real world natural data. Based on the application users, the vocabulary and the style of the model's input may greatly vary. This emphasizes the need for a model agnostic test dataset, which consists of various corruptions that are natural to appear in the wild. Models trained and tested on such benchmark datasets, will be more robust against real world data. However, such data sets are not easily available. In this work, we address this problem, by extending the benchmark datasets along naturally occurring corruptions such as Spelling Errors, Text Noise and Synonyms and making them publicly available. Through extensive experiments, we compare random and targeted corruption strategies using Local Interpretable Model-Agnostic Explanations(LIME). We report the vulnerabilities in two popular text classification models along these corruptions and also find that targeted corruptions can expose vulnerabilities of a model better than random choices in most cases.