Differentially Private Block-wise Gradient Shuffle for Deep Learning
This work addresses efficiency and privacy concerns in deep learning for applications requiring data protection, though it builds on existing private deep learning literature.
The paper tackles the problem of slow training times in differentially private deep learning by introducing DP-BloGS, which uses gradient shuffling and block-wise techniques to achieve training times close to non-private methods while maintaining privacy and utility similar to DP-SGD, with results showing significant resistance to data extraction.
Traditional Differentially Private Stochastic Gradient Descent (DP-SGD) introduces statistical noise on top of gradients drawn from a Gaussian distribution to ensure privacy. This paper introduces the novel Differentially Private Block-wise Gradient Shuffle (DP-BloGS) algorithm for deep learning. BloGS builds off of existing private deep learning literature, but makes a definitive shift by taking a probabilistic approach to gradient noise introduction through shuffling modeled after information theoretic privacy analyses. The theoretical results presented in this paper show that the combination of shuffling, parameter-specific block size selection, batch layer clipping, and gradient accumulation allows DP-BloGS to achieve training times close to that of non-private training while maintaining similar privacy and utility guarantees to DP-SGD. DP-BloGS is found to be significantly more resistant to data extraction attempts than DP-SGD. The theoretical results are validated by the experimental findings.