Semantic Robustness Certification for Vision-Language Models
For practitioners deploying VLMs in real-world applications, this work provides a practical certification method against semantic distribution shifts, addressing a gap in existing certification that focuses on geometric or pixel-level changes.
This paper proposes the first framework to certify robustness of vision-language models under semantic-level transformations (e.g., shape, size, style) without requiring additional data per variation. The framework uses text prompts as semantic proxies and characterizes decision boundaries in closed form to certify extent intervals where predictions remain unchanged.
Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.