Randomized Smoothing under Attack: How Good is it in Pratice?
This work challenges the assumed robustness of a popular defense method, highlighting practical vulnerabilities for machine learning security applications.
The paper questions the practical effectiveness of randomized smoothing as a defense against black-box adversarial attacks, finding a major mismatch between theoretical certification and real-world attack settings.
Randomized smoothing is a recent and celebrated solution to certify the robustness of any classifier. While it indeed provides a theoretical robustness against adversarial attacks, the dimensionality of current classifiers necessarily imposes Monte Carlo approaches for its application in practice. This paper questions the effectiveness of randomized smoothing as a defense, against state of the art black-box attacks. This is a novel perspective, as previous research works considered the certification as an unquestionable guarantee. We first formally highlight the mismatch between a theoretical certification and the practice of attacks on classifiers. We then perform attacks on randomized smoothing as a defense. Our main observation is that there is a major mismatch in the settings of the RS for obtaining high certified robustness or when defeating black box attacks while preserving the classifier accuracy.