Asynchronous Decentralized Learning of a Neural Network
This work addresses communication efficiency in decentralized machine learning, but it is incremental as it builds on existing frameworks and methods.
The paper tackles the problem of communication bottlenecks in decentralized neural network training by proposing an asynchronous decentralized learning algorithm (dSSFN) based on ARock, which reduces communication overhead and increases learning speed, showing competitive performance compared to synchronous methods, especially in sparse networks.
In this work, we exploit an asynchronous computing framework namely ARock to learn a deep neural network called self-size estimating feedforward neural network (SSFN) in a decentralized scenario. Using this algorithm namely asynchronous decentralized SSFN (dSSFN), we provide the centralized equivalent solution under certain technical assumptions. Asynchronous dSSFN relaxes the communication bottleneck by allowing one node activation and one side communication, which reduces the communication overhead significantly, consequently increasing the learning speed. We compare asynchronous dSSFN with traditional synchronous dSSFN in the experimental results, which shows the competitive performance of asynchronous dSSFN, especially when the communication network is sparse.