PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications
This addresses a critical security gap for NLP applications, where Trojan detection was previously lacking, though it is incremental as it adapts image-domain methods to NLP.
The paper tackles the problem of detecting Trojan attacks in NLP models by proposing a perturbation sensitivity-based framework that analyzes model output deviations to identify backdoors, achieving effective detection on both a custom dataset and the public TrojAI dataset with a lightweight variant reducing detection time while maintaining rates.
Deep Neural Networks (DNNs) have been shown to be susceptible to Trojan attacks. Neural Trojan is a type of targeted poisoning attack that embeds the backdoor into the victim and is activated by the trigger in the input space. The increasing deployment of DNNs in critical systems and the surge of outsourcing DNN training (which makes Trojan attack easier) makes the detection of Trojan attacks necessary. While Neural Trojan detection has been studied in the image domain, there is a lack of solutions in the NLP domain. In this paper, we propose a model-level Trojan detection framework by analyzing the deviation of the model output when we introduce a specially crafted perturbation to the input. Particularly, we extract the model's responses to perturbed inputs as the `signature' of the model and train a meta-classifier to determine if a model is Trojaned based on its signature. We demonstrate the effectiveness of our proposed method on both a dataset of NLP models we create and a public dataset of Trojaned NLP models from TrojAI. Furthermore, we propose a lightweight variant of our detection method that reduces the detection time while preserving the detection rates.