How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning
This work addresses the challenge of optimizing generalization for model owners in privacy-preserving distributed learning, though it is incremental as it builds on existing federated learning methods with a focus on client grouping.
The paper tackles the problem of data heterogeneity in cross-silo federated learning by analyzing when client collaboration improves generalization, proposing a hierarchical clustering-based scheme (HCCT) that partitions clients into groups based on data similarity and size, and showing through simulations that HCCT outperforms baselines in generalization performance.
Federated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model's generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved only by collaborating with other clients that have more training data and similar data distribution. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios.