Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent
This work addresses privacy issues for individuals contributing data to decentralized ML training, but it appears incremental as it builds on existing differential privacy and SGD methods.
The paper tackles the problem of privacy concerns in decentralized machine learning by proposing a decentralized differentially private stochastic gradient descent algorithm without replacement, providing both privacy and convergence analysis and demonstrating its effectiveness through experiments.
While machine learning has achieved remarkable results in a wide variety of domains, the training of models often requires large datasets that may need to be collected from different individuals. As sensitive information may be contained in the individual's dataset, sharing training data may lead to severe privacy concerns. Therefore, there is a compelling need to develop privacy-aware machine learning methods, for which one effective approach is to leverage the generic framework of differential privacy. Considering that stochastic gradient descent (SGD) is one of the most commonly adopted methods for large-scale machine learning problems, a decentralized differentially private SGD algorithm is proposed in this work. Particularly, we focus on SGD without replacement due to its favorable structure for practical implementation. Both privacy and convergence analysis are provided for the proposed algorithm. Finally, extensive experiments are performed to demonstrate the effectiveness of the proposed method.