Corrupting Neuron Explanations of Deep Visual Features
This work highlights critical vulnerabilities in explainability methods, raising concerns for their trustworthiness in safety and fairness applications, and is incremental as it builds on existing robustness analyses.
The authors tackled the robustness of neuron explanation methods in deep neural networks, showing that random noise can change the assigned concepts of up to 28% of neurons in deeper layers, and their novel algorithm can manipulate explanations for over 80% of neurons by poisoning less than 10% of probing data.
The inability of DNNs to explain their black-box behavior has led to a recent surge of explainability methods. However, there are growing concerns that these explainability methods are not robust and trustworthy. In this work, we perform the first robustness analysis of Neuron Explanation Methods under a unified pipeline and show that these explanations can be significantly corrupted by random noises and well-designed perturbations added to their probing data. We find that even adding small random noise with a standard deviation of 0.02 can already change the assigned concepts of up to 28% neurons in the deeper layers. Furthermore, we devise a novel corruption algorithm and show that our algorithm can manipulate the explanation of more than 80% neurons by poisoning less than 10% of probing data. This raises the concern of trusting Neuron Explanation Methods in real-life safety and fairness critical applications.