What Do We Mean by Generalization in Federated Learning?
This work addresses a foundational issue in federated learning for researchers, though it is incremental as it builds on existing concepts without major breakthroughs.
The authors tackled the problem of defining generalization in federated learning by proposing a framework to separate performance gaps from unseen client data and distributions, and they observed differences in behavior across datasets, suggesting that synthesis strategies matter for realistic simulations.
Federated learning data is drawn from a distribution of distributions: clients are drawn from a meta-distribution, and their data are drawn from local data distributions. Thus generalization studies in federated learning should separate performance gaps from unseen client data (out-of-sample gap) from performance gaps from unseen client distributions (participation gap). In this work, we propose a framework for disentangling these performance gaps. Using this framework, we observe and explain differences in behavior across natural and synthetic federated datasets, indicating that dataset synthesis strategy can be important for realistic simulations of generalization in federated learning. We propose a semantic synthesis strategy that enables realistic simulation without naturally-partitioned data. Informed by our findings, we call out community suggestions for future federated learning works.