DP-$λ$CGD: Efficient Noise Correlation for Differentially Private Model Training
This work addresses memory efficiency for differentially private model training, which is an incremental improvement over existing methods.
The paper tackles the memory overhead of existing differentially private training methods by proposing a new noise correlation strategy that correlates noise only with the previous iteration and uses pseudorandom noise regeneration, requiring no extra memory. It shows minimal computational overhead and empirically improves accuracy over standard DP-SGD.
Differentially private stochastic gradient descent (DP-SGD) is the gold standard for training machine learning models with formal differential privacy guarantees. Several recent extensions improve its accuracy by introducing correlated noise across training iterations. Matrix factorization mechanisms are a prominent example, but they correlate noise across many iterations and require storing previously added noise vectors, leading to substantial memory overhead in some settings. In this work, we propose a new noise correlation strategy that correlates noise only with the immediately preceding iteration and cancels a controlled portion of it. Our method relies on noise regeneration using a pseudorandom noise generator, eliminating the need to store past noise. As a result, it requires no additional memory beyond standard DP-SGD. We show that the computational overhead is minimal and empirically demonstrate improved accuracy over DP-SGD.