Robust and Efficient Aggregation for Distributed Learning
This work addresses the trade-off between robustness and sample efficiency in distributed learning, which is crucial for applications like federated learning where data privacy and security are concerns, though it appears incremental as it builds on existing robust aggregation techniques.
The paper tackles the problem of robust aggregation in distributed learning, which is susceptible to outliers and malicious agents, by developing new aggregation schemes that achieve statistical efficiency and robustness, reducing the required agent participation rates compared to existing robust methods.
Distributed learning paradigms, such as federated and decentralized learning, allow for the coordination of models across a collection of agents, and without the need to exchange raw data. Instead, agents compute model updates locally based on their available data, and subsequently share the update model with a parameter server or their peers. This is followed by an aggregation step, which traditionally takes the form of a (weighted) average. Distributed learning schemes based on averaging are known to be susceptible to outliers. A single malicious agent is able to drive an averaging-based distributed learning algorithm to an arbitrarily poor model. This has motivated the development of robust aggregation schemes, which are based on variations of the median and trimmed mean. While such procedures ensure robustness to outliers and malicious behavior, they come at the cost of significantly reduced sample efficiency. This means that current robust aggregation schemes require significantly higher agent participation rates to achieve a given level of performance than their mean-based counterparts in non-contaminated settings. In this work we remedy this drawback by developing statistically efficient and robust aggregation schemes for distributed learning.