DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning
This work addresses the challenge of making DP training more practical and stable for machine learning practitioners, though it is incremental as it builds on existing adaptive optimization methods.
The paper tackles the problem of inefficient and hyperparameter-sensitive differentially private (DP) training by proposing DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. It achieves competitive or superior accuracy compared to DP-SGD on benchmarks like CIFAR-10, ImageNet, and transformer fine-tuning, with proven optimal convergence rates.
Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.