From Privacy to Generalization: Linear Max-Information Bounds for DP-SGD
Provides theoretical justification for generalization of DP-SGD, a key algorithm for privacy-preserving deep learning, addressing a persistent open problem.
This work proves a finite-sample bound on the approximate max-information of DP-SGD that scales at most linearly with dataset size, and derives explicit generalization bounds for DP-SGD-trained models with complexity controlled by hyperparameters.
Understanding the relationship between generalization and privacy remains a central challenge in modern machine learning theory, particularly for deep networks trained by variants of differentially private stochastic gradient descent (DP-SGD). In this work we make progress on this persistent open problem by proving a finite-sample bound on the approximate max-information of DP-SGD that exhibits scaling properties comparable with (Dwork et al, 2015)'s classic result for $ε$-differentially private algorithms, namely at most linear in the dataset size. From our result we obtain a general-purpose PAC-Bayes generalization bound in which the necessary prior distribution can be learned by DP-SGD, as well as a generalization bound for DP-SGD-trained models themselves, with a complexity term that is fully explicit and controlled by the optimization hyperparameters.