Federated Asymptotics: a model to compare federated learning algorithms
This work provides a theoretical model to guide practical algorithmic development in federated learning, addressing the need for better comparison tools in this domain, though it is incremental as it builds on existing asymptotic methods.
The authors tackled the problem of comparing federated learning algorithms by proposing an asymptotic framework that models federated learning as a multi-criterion objective, and found that Federated Averaging with client fine-tuning achieves the same asymptotic risk as more complex methods like meta-learning and outperforms non-personalized Federated Averaging, with experiments on EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets corroborating these predictions.
We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client's loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools make fairly precise predictions about the benefits of personalization and information sharing in federated scenarios -- at least in our (stylized) model -- including that Federated Averaging with simple client fine-tuning achieves the same asymptotic risk as the more intricate meta-learning and proximal-regularized approaches and outperforming Federated Averaging without personalization. We evaluate these predictions on federated versions of the EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets, where the experiments corroborate the theoretical predictions, suggesting such frameworks may provide a useful guide to practical algorithmic development.