Resource-Aware Aggregation and Sparsification in Heterogeneous Ensemble Federated Learning
This work addresses computational heterogeneity in federated learning for clients with diverse capacities, representing an incremental advancement in ensemble-based FL methods.
The paper tackles the problem of federated learning convergence hindered by system heterogeneity and communication inefficiency by proposing SHEFL, a global ensemble-based framework that allocates models based on client resources and uses dynamic sparsification, resulting in significant improvements in accuracy and stability over existing methods.
Federated learning (FL) enables distributed training with private client data, but its convergence is hindered by system heterogeneity under realistic communication scenarios. Most FL schemes addressing system heterogeneity utilize global pruning or ensemble distillation, yet often overlook typical constraints required for communication efficiency. Meanwhile, deep ensembles can aggregate predictions from individually trained models to improve performance, but current ensemble-based FL methods fall short in fully capturing diversity of model predictions. In this work, we propose \textbf{SHEFL}, a global ensemble-based FL framework suited for clients with diverse computational capacities. We allocate different numbers of global models to clients based on their available resources. We introduce a novel aggregation scheme that mitigates the training bias between clients and dynamically adjusts the sparsification ratio across clients to reduce the computational burden of training deep ensembles. Extensive experiments demonstrate that our method effectively addresses computational heterogeneity, significantly improving accuracy and stability compared to existing approaches.