Bounding Training Data Reconstruction in DP-SGD
This work addresses the fine-tuning of privacy parameters in DP-SGD for practitioners, offering insights beyond standard differential privacy guarantees to better control against reconstruction attacks.
The paper tackles the problem of bounding training data reconstruction attacks in DP-SGD, showing that less noise can improve utility while still protecting against reconstruction, and demonstrates that different DP-SGD settings with the same differential privacy guarantees lead to significantly different reconstruction success rates.
Differentially private training offers a protection which is usually interpreted as a guarantee against membership inference attacks. By proxy, this guarantee extends to other threats like reconstruction attacks attempting to extract complete training examples. Recent works provide evidence that if one does not need to protect against membership attacks but instead only wants to protect against training data reconstruction, then utility of private models can be improved because less noise is required to protect against these more ambitious attacks. We investigate this further in the context of DP-SGD, a standard algorithm for private deep learning, and provide an upper bound on the success of any reconstruction attack against DP-SGD together with an attack that empirically matches the predictions of our bound. Together, these two results open the door to fine-grained investigations on how to set the privacy parameters of DP-SGD in practice to protect against reconstruction attacks. Finally, we use our methods to demonstrate that different settings of the DP-SGD parameters leading to the same DP guarantees can result in significantly different success rates for reconstruction, indicating that the DP guarantee alone might not be a good proxy for controlling the protection against reconstruction attacks.