Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee
This addresses the trade-off between privacy and utility in decentralized machine learning, offering a method to improve accuracy for applications like federated learning, though it is incremental as it builds on existing DP frameworks.
The paper tackles the problem of accuracy degradation in decentralized learning with differential privacy by proposing Dyn-D$^2$P, which dynamically adjusts gradient clipping and noise levels based on convergence, enhancing model accuracy while preserving privacy, with experiments showing superiority over fixed-noise methods under strong privacy guarantees.
Most existing decentralized learning methods with differential privacy (DP) guarantee rely on constant gradient clipping bounds and fixed-level DP Gaussian noises for each node throughout the training process, leading to a significant accuracy degradation compared to non-private counterparts. In this paper, we propose a new Dynamic Differentially Private Decentralized learning approach (termed Dyn-D$^2$P) tailored for general time-varying directed networks. Leveraging the Gaussian DP (GDP) framework for privacy accounting, Dyn-D$^2$P dynamically adjusts gradient clipping bounds and noise levels based on gradient convergence. This proposed dynamic noise strategy enables us to enhance model accuracy while preserving the total privacy budget. Extensive experiments on benchmark datasets demonstrate the superiority of Dyn-D$^2$P over its counterparts employing fixed-level noises, especially under strong privacy guarantees. Furthermore, we provide a provable utility bound for Dyn-D$^2$P that establishes an explicit dependency on network-related parameters, with a scaling factor of $1/\sqrt{n}$ in terms of the number of nodes $n$ up to a bias error term induced by gradient clipping. To our knowledge, this is the first model utility analysis for differentially private decentralized non-convex optimization with dynamic gradient clipping bounds and noise levels.