Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
This addresses the challenge of data heterogeneity in federated learning for applications requiring privacy, though it appears incremental as it builds on existing aggregation methods.
The paper tackles the problem of training federated learning models on non-IID data by proposing a divide-and-conquer methodology that uses a weight divergence metric to split neural networks, achieving accuracy comparable to or exceeding state-of-the-art aggregation algorithms like FedProx and FedMA, with reported compute and bandwidth optimizations under certain conditions.
Federated Learning allows training of data stored in distributed devices without the need for centralizing training data, thereby maintaining data privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-and-Conquer training methodology that enables the use of the popular FedAvg aggregation algorithm by overcoming the acknowledged FedAvg limitations in non-IID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained model accuracy at par (and in certain cases exceeding) with numbers achieved by state-of-the-art Aggregation algorithms like FedProx, FedMA, etc. Also, we show that this methodology leads to compute and bandwidth optimizations under certain documented conditions.