AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation
This work addresses the challenge of device heterogeneity in federated learning, offering an incremental improvement for faster and more stable training.
The paper tackles the stale model problem in asynchronous federated learning by proposing AsyncFedED, an adaptive weight aggregation algorithm that uses Euclidean distance and local epochs to account for staleness, resulting in improved convergence rate and model accuracy compared to existing methods.
In an asynchronous federated learning framework, the server updates the global model once it receives an update from a client instead of waiting for all the updates to arrive as in the synchronous setting. This allows heterogeneous devices with varied computing power to train the local models without pausing, thereby speeding up the training process. However, it introduces the stale model problem, where the newly arrived update was calculated based on a set of stale weights that are older than the current global model, which may hurt the convergence of the model. In this paper, we present an asynchronous federated learning framework with a proposed adaptive weight aggregation algorithm, referred to as AsyncFedED. To the best of our knowledge this aggregation method is the first to take the staleness of the arrived gradients, measured by the Euclidean distance between the stale model and the current global model, and the number of local epochs that have been performed, into account. Assuming general non-convex loss functions, we prove the convergence of the proposed method theoretically. Numerical results validate the effectiveness of the proposed AsyncFedED in terms of the convergence rate and model accuracy compared to the existing methods for three considered tasks.