Attacks on multimodal models
It addresses security risks for users of multimodal AI systems, but the work appears incremental as it builds on known attack methods for existing architectures.
The paper investigates vulnerabilities in multimodal models, particularly focusing on how attacks on components like CLIP-ViT can be inherited and generalized, with evaluations showing potential risks in industrial applications.
Today, models capable of working with various modalities simultaneously in a chat format are gaining increasing popularity. Despite this, there is an issue of potential attacks on these models, especially considering that many of them include open-source components. It is important to study whether the vulnerabilities of these components are inherited and how dangerous this can be when using such models in the industry. This work is dedicated to researching various types of attacks on such models and evaluating their generalization capabilities. Modern VLM models (LLaVA, BLIP, etc.) often use pre-trained parts from other models, so the main part of this research focuses on them, specifically on the CLIP architecture and its image encoder (CLIP-ViT) and various patch attack variations for it.