Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering
This work addresses the challenge of maintaining model accuracy while ensuring differential privacy in deep learning, which is incremental by building on existing DPSGD methods.
The paper tackled the problem of accuracy degradation in Differentially Private Stochastic Gradient Descent (DPSGD) due to noise and bias, proposing DP-PMLF to simultaneously mitigate these issues and achieving a significantly enhanced privacy-utility trade-off compared to state-of-the-art variants.
Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to train deep neural networks with formal privacy guarantees. However, the addition of differential privacy (DP) often degrades model accuracy by introducing both noise and bias. Existing techniques typically address only one of these issues, as reducing DP noise can exacerbate clipping bias and vice-versa. In this paper, we propose a novel method, \emph{DP-PMLF}, which integrates per-sample momentum with a low-pass filtering strategy to simultaneously mitigate DP noise and clipping bias. Our approach uses per-sample momentum to smooth gradient estimates prior to clipping, thereby reducing sampling variance. It further employs a post-processing low-pass filter to attenuate high-frequency DP noise without consuming additional privacy budget. We provide a theoretical analysis demonstrating an improved convergence rate under rigorous DP guarantees, and our empirical evaluations reveal that DP-PMLF significantly enhances the privacy-utility trade-off compared to several state-of-the-art DPSGD variants.