PID: Prompt-Independent Data Protection Against Latent Diffusion Models
This addresses privacy concerns for individuals whose images are vulnerable to unauthorized AI model training, offering an incremental improvement by focusing on the visual encoder to enhance defense robustness.
The paper tackles the problem of protecting personal images from misuse by Latent Diffusion Models (LDent Diffusion Models) through few-shot fine-tuning, showing that existing defenses fail when textual prompts differ between protectors and exploiters, and proposes a prompt-independent defense method that acts as a strong privacy shield with less computational power.
The few-shot fine-tuning of Latent Diffusion Models (LDMs) has enabled them to grasp new concepts from a limited number of images. However, given the vast amount of personal images accessible online, this capability raises critical concerns about civil privacy. While several previous defense methods have been developed to prevent such misuse of LDMs, they typically assume that the textual prompts used by data protectors exactly match those employed by data exploiters. In this paper, we first empirically demonstrate that breaking this assumption, i.e., in cases where discrepancies exist between the textual conditions used by protectors and exploiters, could substantially reduce the effectiveness of these defenses. Furthermore, considering the visual encoder's independence from textual prompts, we delve into the visual encoder and thoroughly investigate how manipulating the visual encoder affects the few-shot fine-tuning process of LDMs. Drawing on these insights, we propose a simple yet effective method called \textbf{Prompt-Independent Defense (PID)} to safeguard privacy against LDMs. We show that PID can act as a strong privacy shield on its own while requiring significantly less computational power. We believe our studies, along with the comprehensive understanding and new defense method, provide a notable advance toward reliable data protection against LDMs.