Improved Generalization Bounds for Communication Efficient Federated Learning
It addresses communication efficiency for federated learning systems, particularly in non-iid data settings, with incremental improvements based on theoretical analysis.
This paper tackles the problem of high communication costs in federated learning by deriving tighter generalization bounds and proposing a novel algorithm, FedALS, which adapts aggregation frequencies for different model parts, reducing communication by up to 50% in non-iid scenarios.
This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning. We first characterize a tighter generalization bound for one-round federated learning based on local clients' generalizations and heterogeneity of data distribution (non-iid scenario). We also characterize a generalization bound in R-round federated learning and its relation to the number of local updates (local stochastic gradient descents (SGDs)). Then, based on our generalization bound analysis and our representation learning interpretation of this analysis, we show for the first time that less frequent aggregations, hence more local updates, for the representation extractor (usually corresponds to initial layers) leads to the creation of more generalizable models, particularly for non-iid scenarios. We design a novel Federated Learning with Adaptive Local Steps (FedALS) algorithm based on our generalization bound and representation learning analysis. FedALS employs varying aggregation frequencies for different parts of the model, so reduces the communication cost. The paper is followed with experimental results showing the effectiveness of FedALS.