Data Heterogeneity Differential Privacy: From Theory to Algorithm
This work addresses the challenge of balancing privacy and utility in machine learning for applications requiring data protection, though it is incremental as it builds on existing DP-SGD methods.
The paper tackles the problem of improving differential privacy in stochastic gradient descent by accounting for data heterogeneity, proposing a new algorithm (PIDP-SGD) that reduces noise injection for less influential data instances, which leads to significant performance gains as shown in theoretical and experimental results.
Traditionally, the random noise is equally injected when training with different data instances in the field of differential privacy (DP). In this paper, we first give sharper excess risk bounds of DP stochastic gradient descent (SGD) method. Considering most of the previous methods are under convex conditions, we use Polyak-Łojasiewicz condition to relax it in this paper. Then, after observing that different training data instances affect the machine learning model to different extent, we consider the heterogeneity of training data and attempt to improve the performance of DP-SGD from a new perspective. Specifically, by introducing the influence function (IF), we quantitatively measure the contributions of various training data on the final machine learning model. If the contribution made by a single data instance is so little that attackers cannot infer anything from the model, we do not add noise when training with it. Based on this observation, we design a `Performance Improving' DP-SGD algorithm: PIDP-SGD. Theoretical and experimental results show that our proposed PIDP-SGD improves the performance significantly.