Randomized Smoothing Meets Vision-Language Models
This work addresses robustness certification for state-of-the-art vision-language models, making it feasible for applications like content moderation and service-robot commands, but it is incremental as it extends existing randomized smoothing techniques to a new model type.
The paper tackled the problem of applying randomized smoothing for robustness certification to generative vision-language models, which output sequences rather than labels, by connecting them to an oracle classification task and developing theory to associate sample numbers with robustness radius. The result showed that 2 to 3 orders of magnitude fewer samples suffice for certification with minimal loss, validated against adversarial attacks.
Randomized smoothing (RS) is one of the prominent techniques to ensure the correctness of machine learning models, where point-wise robustness certificates can be derived analytically. While RS is well understood for classification, its application to generative models is unclear, since their outputs are sequences rather than labels. We resolve this by connecting generative outputs to an oracle classification task and showing that RS can still be enabled: the final response can be classified as a discrete action (e.g., service-robot commands in VLAs), as harmful vs. harmless (content moderation or toxicity detection in VLMs), or even applying oracles to cluster answers into semantically equivalent ones. Provided that the error rate for the oracle classifier comparison is bounded, we develop the theory that associates the number of samples with the corresponding robustness radius. We further derive improved scaling laws analytically relating the certified radius and accuracy to the number of samples, showing that the earlier result of 2 to 3 orders of magnitude fewer samples sufficing with minimal loss remains valid even under weaker assumptions. Together, these advances make robustness certification both well-defined and computationally feasible for state-of-the-art VLMs, as validated against recent jailbreak-style adversarial attacks.