Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Security
This addresses privacy preservation in deep learning for applications requiring high model accuracy, though it is incremental as it builds on existing DP methods.
The paper tackles the problem of preserving model utility in differentially private deep learning training by introducing GReDP, a method that reduces noise scale by half compared to DPSGD while keeping gradient information intact, achieving consistent improvements across models and settings.
Deep learning models have been extensively adopted in various regions due to their ability to represent hierarchical features, which highly rely on the training set and procedures. Thus, protecting the training process and deep learning algorithms is paramount in privacy preservation. Although Differential Privacy (DP) as a powerful cryptographic primitive has achieved satisfying results in deep learning training, the existing schemes still fall short in preserving model utility, i.e., they either invoke a high noise scale or inevitably harm the original gradients. To address the above issues, in this paper, we present a more robust and provably secure approach for differentially private training called GReDP. Specifically, we compute the model gradients in the frequency domain and adopt a new approach to reduce the noise level. Unlike previous work, our GReDP only requires half of the noise scale compared to DPSGD [1] while keeping all the gradient information intact. We present a detailed analysis of our method both theoretically and empirically. The experimental results show that our GReDP works consistently better than the baselines on all models and training settings.