How the Softmax Output is Misleading for Evaluating the Strength of Adversarial Examples
This addresses a critical issue for machine learning practitioners in security and robustness, but it is incremental as it highlights a flaw in a common evaluation practice rather than introducing a new solution.
The paper tackles the problem of using softmax output to evaluate adversarial example strength, showing that it can be misleading and easily tricked by existing generation methods, leading to inaccurate assessments.
Even before deep learning architectures became the de facto models for complex computer vision tasks, the softmax function was, given its elegant properties, already used to analyze the predictions of feedforward neural networks. Nowadays, the output of the softmax function is also commonly used to assess the strength of adversarial examples: malicious data points designed to fail machine learning models during the testing phase. However, in this paper, we show that it is possible to generate adversarial examples that take advantage of some properties of the softmax function, leading to undesired outcomes when interpreting the strength of the adversarial examples at hand. Specifically, we argue that the output of the softmax function is a poor indicator when the strength of an adversarial example is analyzed and that this indicator can be easily tricked by already existing methods for adversarial example generation.