Device Heterogeneity in Federated Learning: A Superquantile Approach
This work addresses the challenge of non-conforming client devices in federated learning, offering a method to improve robustness in distributed AI systems, though it appears incremental as it builds upon existing federated averaging techniques.
The authors tackled the problem of device heterogeneity in federated learning by proposing a superquantile-based framework, which achieved convergence to a stationary point and was validated through numerical experiments on neural networks and linear models across computer vision and NLP tasks.
We propose a federated learning framework to handle heterogeneous client devices which do not conform to the population data distribution. The approach hinges upon a parameterized superquantile-based objective, where the parameter ranges over levels of conformity. We present an optimization algorithm and establish its convergence to a stationary point. We show how to practically implement it using secure aggregation by interleaving iterations of the usual federated averaging method with device filtering. We conclude with numerical experiments on neural networks as well as linear models on tasks from computer vision and natural language processing.