Skeletonization-Based Adversarial Perturbations on Large Vision Language Model's Mathematical Text Recognition
This work addresses vulnerabilities in large vision-language models for mathematical text recognition, with practical implications for real-world applications, though it is incremental in focusing on a specific attack method.
The paper tackles the problem of evaluating visual capabilities in foundation models by introducing a skeletonization-based adversarial attack method that targets mathematical text images, demonstrating its effectiveness on ChatGPT with detailed character and semantic change analysis.
This work explores the visual capabilities and limitations of foundation models by introducing a novel adversarial attack method utilizing skeletonization to reduce the search space effectively. Our approach specifically targets images containing text, particularly mathematical formula images, which are more challenging due to their LaTeX conversion and intricate structure. We conduct a detailed evaluation of both character and semantic changes between original and adversarially perturbed outputs to provide insights into the models' visual interpretation and reasoning abilities. The effectiveness of our method is further demonstrated through its application to ChatGPT, which shows its practical implications in real-world scenarios.