VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models
This addresses security risks in clinical AI applications, though it is incremental as it applies existing adversarial methods to a new domain.
The paper tackles the lack of systematic security evaluations for medical vision-language models by introducing VSF-Med, a vulnerability-scoring framework that generates adversarial variants and reports mean z-score shifts of up to 0.90σ for attack persistence across state-of-the-art models.
Vision Language Models (VLMs) hold great promise for streamlining labour-intensive medical imaging workflows, yet systematic security evaluations in clinical settings remain scarce. We introduce VSF--Med, an end-to-end vulnerability-scoring framework for medical VLMs that unites three novel components: (i) a rich library of sophisticated text-prompt attack templates targeting emerging threat vectors; (ii) imperceptible visual perturbations calibrated by structural similarity (SSIM) thresholds to preserve clinical realism; and (iii) an eight-dimensional rubric evaluated by two independent judge LLMs, whose raw scores are consolidated via z-score normalization to yield a 0--32 composite risk metric. Built entirely on publicly available datasets and accompanied by open-source code, VSF--Med synthesizes over 30,000 adversarial variants from 5,000 radiology images and enables reproducible benchmarking of any medical VLM with a single command. Our consolidated analysis reports mean z-score shifts of $0.90σ$ for persistence-of-attack-effects, $0.74σ$ for prompt-injection effectiveness, and $0.63σ$ for safety-bypass success across state-of-the-art VLMs. Notably, Llama-3.2-11B-Vision-Instruct exhibits a peak vulnerability increase of $1.29σ$ for persistence-of-attack-effects, while GPT-4o shows increases of $0.69σ$ for that same vector and $0.28σ$ for prompt-injection attacks.