Privacy Backdoors: Stealing Data with Corrupted Pretrained Models
This work exposes a critical supply chain vulnerability in machine learning privacy, affecting practitioners who download and finetune models from untrusted sources.
The paper demonstrates that tampering with pretrained models can create privacy backdoors, allowing attackers to reconstruct individual finetuning samples with guaranteed success, and shows these backdoors enable tight privacy attacks even on models trained with differential privacy.
Practitioners commonly download pretrained machine learning models from open repositories and finetune them to fit specific applications. We show that this practice introduces a new risk of privacy backdoors. By tampering with a pretrained model's weights, an attacker can fully compromise the privacy of the finetuning data. We show how to build privacy backdoors for a variety of models, including transformers, which enable an attacker to reconstruct individual finetuning samples, with a guaranteed success! We further show that backdoored models allow for tight privacy attacks on models trained with differential privacy (DP). The common optimistic practice of training DP models with loose privacy guarantees is thus insecure if the model is not trusted. Overall, our work highlights a crucial and overlooked supply chain attack on machine learning privacy.