Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
This work addresses data privacy concerns for users and developers of MLLMs, but it is incremental as it extends existing MIA methods to multimodal contexts.
The paper tackled the problem of evaluating text-based membership inference attacks (MIAs) on large multimodal models (MLLMs) to understand training-data leakage, finding that in in-distribution settings, logit-based MIAs perform comparably with a slight advantage for vision-and-text inputs, while in out-of-distribution settings, visual inputs mask membership signals.
Large Multimodal Language Models (MLLMs) are emerging as one of the foundational tools in an expanding range of applications. Consequently, understanding training-data leakage in these systems is increasingly critical. Log-probability-based membership inference attacks (MIAs) have become a widely adopted approach for assessing data exposure in large language models (LLMs), yet their effect in MLLMs remains unclear. We present the first comprehensive evaluation of extending these text-based MIA methods to multimodal settings. Our experiments under vision-and-text (V+T) and text-only (T-only) conditions across the DeepSeek-VL and InternVL model families show that in in-distribution settings, logit-based MIAs perform comparably across configurations, with a slight V+T advantage. Conversely, in out-of-distribution settings, visual inputs act as regularizers, effectively masking membership signals.