Cross-Silo Federated Learning Across Divergent Domains with Iterative Parameter Alignment
This addresses the challenge of collaborative learning in cross-silo settings with disparate data, offering a flexible framework for improved generalization, though it appears incremental as it builds on existing federated learning methods.
The paper tackles the problem of federated learning across clients with divergent domains, where existing methods struggle to converge and produce identical models, by proposing Iterative Parameter Alignment to learn separate models for each client while optimizing a common objective, achieving competitive results and robustness in divergent scenarios.
Learning from the collective knowledge of data dispersed across private sources can provide neural networks with enhanced generalization capabilities. Federated learning, a method for collaboratively training a machine learning model across remote clients, achieves this by combining client models via the orchestration of a central server. However, current approaches face two critical limitations: i) they struggle to converge when client domains are sufficiently different, and ii) current aggregation techniques produce an identical global model for each client. In this work, we address these issues by reformulating the typical federated learning setup: rather than learning a single global model, we learn N models each optimized for a common objective. To achieve this, we apply a weighted distance minimization to model parameters shared in a peer-to-peer topology. The resulting framework, Iterative Parameter Alignment, applies naturally to the cross-silo setting, and has the following properties: (i) a unique solution for each participant, with the option to globally converge each model in the federation, and (ii) an optional early-stopping mechanism to elicit fairness among peers in collaborative learning settings. These characteristics jointly provide a flexible new framework for iteratively learning from peer models trained on disparate datasets. We find that the technique achieves competitive results on a variety of data partitions compared to state-of-the-art approaches. Further, we show that the method is robust to divergent domains (i.e. disjoint classes across peers) where existing approaches struggle.