Scale-Invariant Adversarial Attack for Evaluating and Enhancing Adversarial Defenses
This addresses the problem of unreliable adversarial evaluation and weak defenses for machine learning practitioners, offering incremental improvements by stabilizing attacks and boosting defense mechanisms.
The paper tackles the vulnerability of standard adversarial attacks like PGD to logit rescaling, proposing a scale-invariant attack (SI-PGD) that uses cosine angles in the penultimate layer to generate stable adversaries, and shows it outperforms existing attacks while also enhancing defenses to achieve state-of-the-art performance.
Efficient and effective attacks are crucial for reliable evaluation of defenses, and also for developing robust models. Projected Gradient Descent (PGD) attack has been demonstrated to be one of the most successful adversarial attacks. However, the effect of the standard PGD attack can be easily weakened by rescaling the logits, while the original decision of every input will not be changed. To mitigate this issue, in this paper, we propose Scale-Invariant Adversarial Attack (SI-PGD), which utilizes the angle between the features in the penultimate layer and the weights in the softmax layer to guide the generation of adversaries. The cosine angle matrix is used to learn angularly discriminative representation and will not be changed with the rescaling of logits, thus making SI-PGD attack to be stable and effective. We evaluate our attack against multiple defenses and show improved performance when compared with existing attacks. Further, we propose Scale-Invariant (SI) adversarial defense mechanism based on the cosine angle matrix, which can be embedded into the popular adversarial defenses. The experimental results show the defense method with our SI mechanism achieves state-of-the-art performance among multi-step and single-step defenses.