From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization
This addresses performance degradation in federated learning systems due to asynchronous updates, offering a solution for distributed machine learning applications, though it appears incremental as it builds on existing federated optimization methods.
The paper tackles the problem of step asynchronism in federated optimization, which degrades model accuracy under non-i.i.d. data, by proposing FedaGrac, an algorithm that calibrates local updates to a predictive global orientation, resulting in improved convergence rates and enhanced final accuracy.
In the setting of federated optimization, where a global model is aggregated periodically, step asynchronism occurs when participants conduct model training by efficiently utilizing their computational resources. It is well acknowledged that step asynchronism leads to objective inconsistency under non-i.i.d. data, which degrades the model's accuracy. To address this issue, we propose a new algorithm FedaGrac, which calibrates the local direction to a predictive global orientation. Taking advantage of the estimated orientation, we guarantee that the aggregated model does not excessively deviate from the global optimum while fully utilizing the local updates of faster nodes. We theoretically prove that FedaGrac holds an improved order of convergence rate than the state-of-the-art approaches and eliminates the negative effect of step asynchronism. Empirical results show that our algorithm accelerates the training and enhances the final accuracy.