Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning
This addresses efficiency and adaptability issues in federated learning for scenarios with heterogeneous client data, though it is an incremental improvement over existing client selection methods.
The paper tackles the challenge of training global models in federated learning with non-IID data and limited client participation by proposing HiCS-FL, a client selection method that clusters clients based on statistical heterogeneity estimated from output layer updates. It achieves faster convergence and drastically reduces computation cost compared to state-of-the-art schemes.
Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.