TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Models
This addresses privacy concerns for users of large-scale multimodal models by enabling identity inference without exposing personal images or incurring high computational costs, though it is an incremental improvement over existing methods.
The paper tackled the problem of identity inference in CLIP models by proposing TUNI, a textual unimodal detector that uses only text data and avoids training shadow models, achieving superior performance over baselines in experiments across various architectures and datasets.
The widespread usage of large-scale multimodal models like CLIP has heightened concerns about the leakage of PII. Existing methods for identity inference in CLIP models require querying the model with full PII, including textual descriptions of the person and corresponding images (e.g., the name and the face photo of the person). However, applying images may risk exposing personal information to target models, as the image might not have been previously encountered by the target model. Additionally, previous MIAs train shadow models to mimic the behaviors of the target model, which incurs high computational costs, especially for large CLIP models. To address these challenges, we propose a textual unimodal detector (TUNI) in CLIP models, a novel technique for identity inference that: 1) only utilizes text data to query the target model; and 2) eliminates the need for training shadow models. Extensive experiments of TUNI across various CLIP model architectures and datasets demonstrate its superior performance over baselines, albeit with only text data.