Backdooring Explainable Machine Learning
This addresses a security vulnerability in explainable AI systems, which is an incremental but important concern for users relying on these methods for trustworthy analysis.
The paper tackles the problem of explainable machine learning methods being manipulated to present unfaithful explanations, demonstrating blinding attacks that disguise ongoing attacks against models and fool explanations, with results including successful red-herring attacks in malware classification.
Explainable machine learning holds great potential for analyzing and understanding learning-based systems. These methods can, however, be manipulated to present unfaithful explanations, giving rise to powerful and stealthy adversaries. In this paper, we demonstrate blinding attacks that can fully disguise an ongoing attack against the machine learning model. Similar to neural backdoors, we modify the model's prediction upon trigger presence but simultaneously also fool the provided explanation. This enables an adversary to hide the presence of the trigger or point the explanation to entirely different portions of the input, throwing a red herring. We analyze different manifestations of such attacks for different explanation types in the image domain, before we resume to conduct a red-herring attack against malware classification.