Introducing Noise in Decentralized Training of Neural Networks
This addresses the challenge of enhancing model performance in decentralized settings for machine learning practitioners, though it is incremental as it builds on known noise injection techniques.
The paper tackled the problem of improving generalization in decentralized neural network training by injecting noise into weights, finding that while it has no positive effect on linear models, it substantially improves model quality for non-linear networks, achieving generalization close to a serial baseline.
It has been shown that injecting noise into the neural network weights during the training process leads to a better generalization of the resulting model. Noise injection in the distributed setup is a straightforward technique and it represents a promising approach to improve the locally trained models. We investigate the effects of noise injection into the neural networks during a decentralized training process. We show both theoretically and empirically that noise injection has no positive effect in expectation on linear models, though. However for non-linear neural networks we empirically show that noise injection substantially improves model quality helping to reach a generalization ability of a local model close to the serial baseline.