DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
This addresses the problem of low utility in privacy-preserving machine learning for practitioners, but it is incremental as it builds on existing DP-SGD methods.
The paper tackles the utility degradation in differentially private stochastic gradient descent (DP-SGD) by proposing DP-LSSGD, which uses Laplacian smoothing to stabilize training and improve generalization for convex and nonconvex models, achieving the same DP guarantee with negligible extra overhead.
Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones. To mitigate this degradation, we propose a DP Laplacian smoothing SGD (DP-LSSGD) to train ML models with differential privacy (DP) guarantees. At the core of DP-LSSGD is the Laplacian smoothing, which smooths out the Gaussian noise used in the Gaussian mechanism. Under the same amount of noise used in the Gaussian mechanism, DP-LSSGD attains the same DP guarantee, but in practice, DP-LSSGD makes training both convex and nonconvex ML models more stable and enables the trained models to generalize better. The proposed algorithm is simple to implement and the extra computational complexity and memory overhead compared with DP-SGD are negligible. DP-LSSGD is applicable to train a large variety of ML models, including DNNs. The code is available at \url{https://github.com/BaoWangMath/DP-LSSGD}.