Backpropagation Clipping for Deep Learning with Differential Privacy
This addresses the need for more accurate differentially private deep learning, though it appears incremental as a variant of existing DP-SGD methods.
The paper tackles the problem of privacy-preserving deep learning by introducing backpropagation clipping, a variant of differentially private stochastic gradient descent that clips inputs and gradients at each layer. The method achieves 98.7% accuracy on MNIST with ε=0.07 and 74% accuracy on CIFAR-10 with ε=3.64, outperforming previous work at lower privacy parameters.
We present backpropagation clipping, a novel variant of differentially private stochastic gradient descent (DP-SGD) for privacy-preserving deep learning. Our approach clips each trainable layer's inputs (during the forward pass) and its upstream gradients (during the backward pass) to ensure bounded global sensitivity for the layer's gradient; this combination replaces the gradient clipping step in existing DP-SGD variants. Our approach is simple to implement in existing deep learning frameworks. The results of our empirical evaluation demonstrate that backpropagation clipping provides higher accuracy at lower values for the privacy parameter $ε$ compared to previous work. We achieve 98.7% accuracy for MNIST with $ε= 0.07$ and 74% accuracy for CIFAR-10 with $ε= 3.64$.