A Low-Rank Defense Method for Adversarial Attack on Diffusion Models
This work addresses the security of diffusion models against adversarial attacks, which is crucial for their practical application, but it is an incremental improvement over existing defense strategies.
The paper tackles the problem of adversarial attacks on Latent Diffusion Models by proposing a defense method called Low-Rank Defense (LoRD), which uses low-rank adaptation modules to detect and defend against adversarial samples, resulting in significantly better defense performance on facial and landscape images compared to baselines.
Recently, adversarial attacks for diffusion models as well as their fine-tuning process have been developed rapidly. To prevent the abuse of these attack algorithms from affecting the practical application of diffusion models, it is critical to develop corresponding defensive strategies. In this work, we propose an efficient defensive strategy, named Low-Rank Defense (LoRD), to defend the adversarial attack on Latent Diffusion Models (LDMs). LoRD introduces the merging idea and a balance parameter, combined with the low-rank adaptation (LoRA) modules, to detect and defend the adversarial samples. Based on LoRD, we build up a defense pipeline that applies the learned LoRD modules to help diffusion models defend against attack algorithms. Our method ensures that the LDM fine-tuned on both adversarial and clean samples can still generate high-quality images. To demonstrate the effectiveness of our approach, we conduct extensive experiments on facial and landscape images, and our method shows significantly better defense performance compared to the baseline methods.