Zero-shot image privacy classification with Vision-Language Models
This work addresses the need for fair evaluation of VLMs in privacy prediction for researchers and practitioners, but it is incremental as it focuses on benchmarking rather than proposing new methods.
The authors tackled the problem of evaluating Vision-Language Models (VLMs) for zero-shot image privacy classification by establishing a benchmark and comparing them to specialized models, finding that VLMs lag in accuracy but show higher robustness to perturbations.
While specialized learning-based models have historically dominated image privacy prediction, the current literature increasingly favours adopting large Vision-Language Models (VLMs) designed for generic tasks. This trend risks overlooking the performance ceiling set by purpose-built models due to a lack of systematic evaluation. To address this problem, we establish a zero-shot benchmark for image privacy classification, enabling a fair comparison. We evaluate the top-3 open-source VLMs, according to a privacy benchmark, using task-aligned prompts and we contrast their performance, efficiency, and robustness against established vision-only and multi-modal methods. Counter-intuitively, our results show that VLMs, despite their resource-intensive nature in terms of high parameter count and slower inference, currently lag behind specialized, smaller models in privacy prediction accuracy. We also find that VLMs exhibit higher robustness to image perturbations.