Gradient Obfuscation Checklist Test Gives a False Sense of Security
This work addresses a critical issue for researchers and practitioners in adversarial machine learning by exposing limitations in current evaluation methods, highlighting an incremental but important refinement in understanding defense mechanisms.
The paper tackles the problem of gradient obfuscation in stochastic defenses against adversarial attacks, showing that a popular checklist test gives a false sense of security by presenting a counterexample where the test fails to detect obfuscation as the main source of robustness.
One popular group of defense techniques against adversarial attacks is based on injecting stochastic noise into the network. The main source of robustness of such stochastic defenses however is often due to the obfuscation of the gradients, offering a false sense of security. Since most of the popular adversarial attacks are optimization-based, obfuscated gradients reduce their attacking ability, while the model is still susceptible to stronger or specifically tailored adversarial attacks. Recently, five characteristics have been identified, which are commonly observed when the improvement in robustness is mainly caused by gradient obfuscation. It has since become a trend to use these five characteristics as a sufficient test, to determine whether or not gradient obfuscation is the main source of robustness. However, these characteristics do not perfectly characterize all existing cases of gradient obfuscation, and therefore can not serve as a basis for a conclusive test. In this work, we present a counterexample, showing this test is not sufficient for concluding that gradient obfuscation is not the main cause of improvements in robustness.