Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning
This work provides a more realistic understanding of information leakage in differentially private machine learning for practitioners and researchers, by showing that theoretical bounds may be overly pessimistic under practical adversary restrictions.
This paper instantiates a hypothetical adversary to establish lower bounds on the probability of distinguishing between models trained on nearly identical datasets in differentially private machine learning. For DP-SGD, the most common method, their lower bounds are tight and match theoretical upper bounds, indicating that stronger assumptions are needed for better upper bounds. However, they find that practical restrictions on adversary capabilities significantly weaken these attacks, suggesting that theoretical differential privacy bounds can be conservative in real-world deployments.
Differentially private (DP) machine learning allows us to train models on private data while limiting data leakage. DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example.If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private. Hence, the purpose of privacy analysis is to upper bound the probability that any adversary could successfully guess which dataset the model was trained on.In our paper, we instantiate this hypothetical adversary in order to establish lower bounds on the probability that this distinguishing game can be won. We use this adversary to evaluate the importance of the adversary capabilities allowed in the privacy analysis of DP training algorithms.For DP-SGD, the most common method for training neural networks with differential privacy, our lower bounds are tight and match the theoretical upper bound. This implies that in order to prove better upper bounds, it will be necessary to make use of additional assumptions. Fortunately, we find that our attacks are significantly weaker when additional (realistic)restrictions are put in place on the adversary's capabilities.Thus, in the practical setting common to many real-world deployments, there is a gap between our lower bounds and the upper bounds provided by the analysis: differential privacy is conservative and adversaries may not be able to leak as much information as suggested by the theoretical bound.