A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via $f$-Divergences
This work provides enhanced privacy guarantees for machine learning practitioners, enabling more efficient training of deep learning models under differential privacy constraints, though it is incremental as it builds on existing RDP and moments accountant methods.
The paper tackles the problem of deriving optimal differential privacy parameters for mechanisms satisfying Rényi differential privacy, using f-divergences, and applies this to the moments accountant framework for stochastic gradient descent, resulting in up to 100 more training iterations for deep learning models under the same privacy budget.
We derive the optimal differential privacy (DP) parameters of a mechanism that satisfies a given level of Rényi differential privacy (RDP). Our result is based on the joint range of two $f$-divergences that underlie the approximate and the Rényi variations of differential privacy. We apply our result to the moments accountant framework for characterizing privacy guarantees of stochastic gradient descent. When compared to the state-of-the-art, our bounds may lead to about 100 more stochastic gradient descent iterations for training deep learning models for the same privacy budget.