Dynamic Differential-Privacy Preserving SGD
This addresses the problem of unstable updates and lower accuracy in DP-SGD for machine learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the performance loss in differentially private stochastic gradient descent (DP-SGD) by proposing dynamic DP-SGD, which adjusts clipping thresholds and noise powers dynamically, resulting in significantly improved model accuracy in strong privacy protection regions compared to vanilla DP-SGD.
The vanilla Differentially-Private Stochastic Gradient Descent (DP-SGD), including DP-Adam and other variants, ensures the privacy of training data by uniformly distributing privacy costs across training steps. The equivalent privacy costs controlled by maintaining the same gradient clipping thresholds and noise powers in each step result in unstable updates and a lower model accuracy when compared to the non-DP counterpart. In this paper, we propose the dynamic DP-SGD (along with dynamic DP-Adam, and others) to reduce the performance loss gap while maintaining privacy by dynamically adjusting clipping thresholds and noise powers while adhering to a total privacy budget constraint. Extensive experiments on a variety of deep learning tasks, including image classification, natural language processing, and federated learning, demonstrate that the proposed dynamic DP-SGD algorithm stabilizes updates and, as a result, significantly improves model accuracy in the strong privacy protection region when compared to the vanilla DP-SGD. We also conduct theoretical analysis to better understand the privacy-utility trade-off with dynamic DP-SGD, as well as to learn why Dynamic DP-SGD can outperform vanilla DP-SGD.