ROSS:RObust decentralized Stochastic learning based on Shapley values
This addresses robust model training in decentralized settings for applications like federated learning, though it appears incremental as it builds on existing methods with a novel weighting scheme.
The paper tackles the problem of data heterogeneity, noise, and poisoning in decentralized learning by proposing ROSS, a robust algorithm using Shapley values to weight cross-gradient information, achieving linear convergence and superior accuracy over state-of-the-art methods.
In the paradigm of decentralized learning, a group of agents collaborate to learn a global model using a distributed dataset without a central server; nevertheless, it is severely challenged by the heterogeneity of the data distribution across the agents. For example, the data may be distributed non-independently and identically, and even be noised or poisoned. To address these data challenges, we propose ROSS, a novel robust decentralized stochastic learning algorithm based on Shapley values, in this paper. Specifically, in each round, each agent aggregates the cross-gradient information from its neighbors, i.e., the derivatives of its local model with respect to the datasets of its neighbors, to update its local model in a momentum like manner, while we innovate in weighting the derivatives according to their contributions measured by Shapley values. We perform solid theoretical analysis to reveal the linear convergence speedup of our ROSS algorithm. We also verify the efficacy of our algorithm through extensive experiments on public datasets. Our results demonstrate that, in face of the above variety of data challenges, our ROSS algorithm have oblivious advantages over existing state-of-the-art proposals in terms of both convergence and prediction accuracy.