Hierarchically Robust Zero-shot Vision-language Models
For practitioners deploying VLMs in safety-critical applications, this work addresses the overlooked issue of hierarchical adversarial robustness, offering a method that generalizes better than existing robust fine-tuning approaches.
Vision-Language Models are vulnerable to adversarial attacks, especially when targeting hierarchical class structures. The proposed hierarchical adversarial fine-tuning framework improves robustness across multiple levels of class hierarchy while maintaining natural performance.
Vision-Language Models (VLMs) can perform zero-shot classification but are susceptible to adversarial attacks. While robust fine-tuning improves their robustness, existing approaches align fixed text embeddings with an image embedding, sacrificing natural performance and robustness. A robustness degradation also occurs when a model faces adversarial attacks targeting superclasses (parent classes, e.g., mammal) in addition to their base (leaf) classes (e.g., cat). Thus, to enhance adversarial robustness and leverage the inherent hierarchical properties of class space, we propose a novel adversarial fine-tuning framework based on hierarchical embeddings and several levels of adversarially robust alignment of image-text modalities. Additional mechanisms place visual embeddings at the desired depth of hierarchy, and we provide a theoretical connection between the depth of embedding in the hierarchy and the maximum viable margin size. Our model naturally realizes several margin sizes, boosting generalization of adversaries for robustification. As various trees with different parent labels can share the same leaf labels, we also consider aligning over multiple trees to boost semantic variety. Experiments across several datasets are performed.