Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent
This work solves a practical limitation in applying differential privacy to deep neural networks, enabling the use of Batch Normalization for improved accuracy, though it is incremental within the DPSGD framework.
The paper tackles the incompatibility of Batch Normalization Layers with Individual Clipping in Differential Private Stochastic Gradient Descent (DPSGD) by introducing Batch Clipping, and addresses sensitivity variations across layers with a new Adaptive Layerwise Clipping method, providing rigorous DP proofs and showing convergence on CIFAR-10 with resnet-18 where previous methods fail.
Each round in Differential Private Stochastic Gradient Descent (DPSGD) transmits a sum of clipped gradients obfuscated with Gaussian noise to a central server which uses this to update a global model which often represents a deep neural network. Since the clipped gradients are computed separately, which we call Individual Clipping (IC), deep neural networks like resnet-18 cannot use Batch Normalization Layers (BNL) which is a crucial component in deep neural networks for achieving a high accuracy. To utilize BNL, we introduce Batch Clipping (BC) where, instead of clipping single gradients as in the orginal DPSGD, we average and clip batches of gradients. Moreover, the model entries of different layers have different sensitivities to the added Gaussian noise. Therefore, Adaptive Layerwise Clipping methods (ALC), where each layer has its own adaptively finetuned clipping constant, have been introduced and studied, but so far without rigorous DP proofs. In this paper, we propose {\em a new ALC and provide rigorous DP proofs for both BC and ALC}. Experiments show that our modified DPSGD with BC and ALC for CIFAR-$10$ with resnet-$18$ converges while DPSGD with IC and ALC does not.