DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation
This addresses a specific optimization problem for users of differentially private machine learning, but it is incremental as it builds on existing DP and Adam methods.
The authors tackled the bias in second moment estimation when using differential privacy with the Adam optimizer, which arises from independent noise addition in gradient computation, and found that correcting this bias significantly improves optimization performance in DP-Adam.
We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.