OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models
This work addresses a critical issue for researchers and practitioners in AI privacy by exposing limitations in current MIA evaluations, potentially leading to more robust privacy-preserving techniques.
The paper tackled the problem of evaluating membership inference attacks (MIA) on large vision-language models by revealing that prior high success rates stemmed from dataset biases rather than true membership detection. It introduced OpenLVLM-MIA, a controlled benchmark with 6,000 images, showing that state-of-the-art MIA methods performed at random chance under unbiased conditions.
OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods converged to random chance under unbiased conditions. By offering a transparent and unbiased benchmark, OpenLVLM-MIA clarifies the current limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.