Normal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction
This work addresses the problem of understanding adversarial samples in relation extraction for AI security researchers, but it is incremental as it builds on existing adversarial analysis efforts.
The paper tackles the vulnerability of neural relation extraction models to adversarial attacks by analyzing differences between normal and adversarial samples using a salience-based method, finding that adversarial perturbations correlate with salience tokens and often involve out-of-training tokens or superficial cues.
Recent neural-based relation extraction approaches, though achieving promising improvement on benchmark datasets, have reported their vulnerability towards adversarial attacks. Thus far, efforts mostly focused on generating adversarial samples or defending adversarial attacks, but little is known about the difference between normal and adversarial samples. In this work, we take the first step to leverage the salience-based method to analyze those adversarial samples. We observe that salience tokens have a direct correlation with adversarial perturbations. We further find the adversarial perturbations are either those tokens not existing in the training set or superficial cues associated with relation labels. To some extent, our approach unveils the characters against adversarial samples. We release an open-source testbed, "DiagnoseAdv" in https://github.com/zjunlp/DiagnoseAdv.