Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks
This work addresses the challenge of fusing models from diverse clients in federated learning, though it is incremental as it builds on existing fusion methods with a new experimental approach.
The paper tackled the problem of neuron interference when fusing heterogeneous neural networks in federated learning, revealing the phenomenon of neuron disturbing and proposing an adaptive model selection (AMS) method that improved robustness to data heterogeneity compared to general fusion and ensemble methods.
Fusing deep learning models trained on separately located clients into a global model in a one-shot communication round is a straightforward implementation of Federated Learning. Although current model fusion methods are shown experimentally valid in fusing neural networks with almost identical architectures, they are rarely theoretically analyzed. In this paper, we reveal the phenomenon of neuron disturbing, where neurons from heterogeneous local models interfere with each other mutually. We give detailed explanations from a Bayesian viewpoint combining the data heterogeneity among clients and properties of neural networks. Furthermore, to validate our findings, we propose an experimental method that excludes neuron disturbing and fuses neural networks via adaptively selecting a local model, called AMS, to execute the prediction according to the input. The experiments demonstrate that AMS is more robust in data heterogeneity than general model fusion and ensemble methods. This implies the necessity of considering neural disturbing in model fusion. Besides, AMS is available for fusing models with varying architectures as an experimental algorithm, and we also list several possible extensions of AMS for future work.