Measuring the Transferability of $\ell_\infty$ Attacks by the $\ell_2$ Norm
This addresses a methodological issue in adversarial machine learning for researchers, offering a more nuanced evaluation standard, though it is incremental in refining existing metrics.
The paper tackles the problem of measuring adversarial attack strength by showing that using only the ℓ∞ norm is insufficient, as ℓ2 distance significantly affects transferability between models, with existing methods achieving 70% to 130% larger ℓ2 distances leading to better performance. It proposes measuring attack strength with both ℓ∞ and ℓ2 norms, supported by experiments on ImageNet with multiple attacks and models.
Deep neural networks could be fooled by adversarial examples with trivial differences to original samples. To keep the difference imperceptible in human eyes, researchers bound the adversarial perturbations by the $\ell_\infty$ norm, which is now commonly served as the standard to align the strength of different attacks for a fair comparison. However, we propose that using the $\ell_\infty$ norm alone is not sufficient in measuring the attack strength, because even with a fixed $\ell_\infty$ distance, the $\ell_2$ distance also greatly affects the attack transferability between models. Through the discovery, we reach more in-depth understandings towards the attack mechanism, i.e., several existing methods attack black-box models better partly because they craft perturbations with 70% to 130% larger $\ell_2$ distances. Since larger perturbations naturally lead to better transferability, we thereby advocate that the strength of attacks should be simultaneously measured by both the $\ell_\infty$ and $\ell_2$ norm. Our proposal is firmly supported by extensive experiments on ImageNet dataset from 7 attacks, 4 white-box models, and 9 black-box models.