Differential Privacy Guarantees for Stochastic Gradient Langevin Dynamics
This work addresses privacy concerns in machine learning for users handling sensitive data, but it is incremental as it builds on existing methods for differential privacy in stochastic settings.
The paper tackled the problem of privacy leakage in noisy stochastic gradient descent by analyzing Rényi divergence dynamics with Langevin diffusions, proving that privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size, which is a significant improvement over previous DP-SGD analyses, and experiments showed practical utility compared to classical DP-SGD libraries.
We analyse the privacy leakage of noisy stochastic gradient descent by modeling Rényi divergence dynamics with Langevin diffusions. Inspired by recent work on non-stochastic algorithms, we derive similar desirable properties in the stochastic setting. In particular, we prove that the privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size, which is a significant improvement over previous DP-SGD analyses. We also extend our analysis to arbitrary sequences of varying step sizes and derive new utility bounds. Last, we propose an implementation and our experiments show the practical utility of our approach compared to classical DP-SGD libraries.