Private Adaptive Gradient Methods for Convex Optimization
This work addresses privacy-preserving optimization for machine learning, offering improved performance over prior methods in high-dimensional settings, though it is incremental in adapting existing adaptive techniques to differential privacy.
The paper tackles the problem of differentially private convex optimization by proposing private variants of adaptive gradient methods like SGD with adaptive stepsizes and AdaGrad, showing that these methods achieve optimal worst-case regret bounds and outperform traditional private SGD in scenarios with non-isotropic gradients.
We study adaptive methods for differentially private convex optimization, proposing and analyzing differentially private variants of a Stochastic Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the AdaGrad algorithm. We provide upper bounds on the regret of both algorithms and show that the bounds are (worst-case) optimal. As a consequence of our development, we show that our private versions of AdaGrad outperform adaptive SGD, which in turn outperforms traditional SGD in scenarios with non-isotropic gradients where (non-private) Adagrad provably outperforms SGD. The major challenge is that the isotropic noise typically added for privacy dominates the signal in gradient geometry for high-dimensional problems; approaches to this that effectively optimize over lower-dimensional subspaces simply ignore the actual problems that varying gradient geometries introduce. In contrast, we study non-isotropic clipping and noise addition, developing a principled theoretical approach; the consequent procedures also enjoy significantly stronger empirical performance than prior approaches.