Gradient Sparsification Can Improve Performance of Differentially-Private Convex Machine Learning
This work addresses the challenge of improving the performance of differentially-private machine learning models, which is relevant for researchers and practitioners working with sensitive data.
This paper explores the use of gradient sparsification to mitigate the negative impact of differential privacy noise on machine learning model performance. They found that for small privacy budgets, compression can improve the performance of privacy-preserving machine learning models, whereas for large privacy budgets, it does not necessarily offer improvement.
We use gradient sparsification to reduce the adverse effect of differential privacy noise on performance of private machine learning models. To this aim, we employ compressed sensing and additive Laplace noise to evaluate differentially-private gradients. Noisy privacy-preserving gradients are used to perform stochastic gradient descent for training machine learning models. Sparsification, achieved by setting the smallest gradient entries to zero, can reduce the convergence speed of the training algorithm. However, by sparsification and compressed sensing, the dimension of communicated gradient and the magnitude of additive noise can be reduced. The interplay between these effects determines whether gradient sparsification improves the performance of differentially-private machine learning models. We investigate this analytically in the paper. We prove that, for small privacy budgets, compression can improve performance of privacy-preserving machine learning models. However, for large privacy budgets, compression does not necessarily improve the performance. Intuitively, this is because the effect of privacy-preserving noise is minimal in large privacy budget regime and thus improvements from gradient sparsification cannot compensate for its slower convergence.