Improving Generalization in Federated Learning by Seeking Flat Minima
This addresses generalization issues in federated learning for applications like vision tasks, but it is incremental as it adapts existing optimization methods to this setting.
The paper tackles the problem of poor generalization in federated learning, especially in heterogeneous scenarios, by linking it to sharp loss minima and showing that using Sharpness-Aware Minimization (SAM) or ASAM locally and averaging stochastic weights (SWA) on the server improves generalization, with empirical results across multiple vision datasets and tasks.
Models trained in federated settings often suffer from degraded performances and fail at generalizing, especially when facing heterogeneous scenarios. In this work, we investigate such behavior through the lens of geometry of the loss and Hessian eigenspectrum, linking the model's lack of generalization capacity to the sharpness of the solution. Motivated by prior studies connecting the sharpness of the loss surface and the generalization gap, we show that i) training clients locally with Sharpness-Aware Minimization (SAM) or its adaptive version (ASAM) and ii) averaging stochastic weights (SWA) on the server-side can substantially improve generalization in Federated Learning and help bridging the gap with centralized models. By seeking parameters in neighborhoods having uniform low loss, the model converges towards flatter minima and its generalization significantly improves in both homogeneous and heterogeneous scenarios. Empirical results demonstrate the effectiveness of those optimizers across a variety of benchmark vision datasets (e.g. CIFAR10/100, Landmarks-User-160k, IDDA) and tasks (large scale classification, semantic segmentation, domain generalization).