Consensus Driven Learning
This addresses the problem of high data and computation requirements for large neural networks by enabling efficient distributed training without centralized coordination, though it is incremental as it builds on existing consensus algorithms.
The paper tackles the challenge of training neural networks on distributed devices with local datasets by proposing a decentralized, asynchronous coordination method using Distributed Averaging Consensus, which enables learning on biased datasets and under intermittent communication failures, as demonstrated on MNIST, Fashion MNIST, and CIFAR10 datasets.
As the complexity of our neural network models grow, so too do the data and computation requirements for successful training. One proposed solution to this problem is training on a distributed network of computational devices, thus distributing the computational and data storage loads. This strategy has already seen some adoption by the likes of Google and other companies. In this paper we propose a new method of distributed, decentralized learning that allows a network of computation nodes to coordinate their training using asynchronous updates over an unreliable network while only having access to a local dataset. This is achieved by taking inspiration from Distributed Averaging Consensus algorithms to coordinate the various nodes. Sharing the internal model instead of the training data allows the original raw data to remain with the computation node. The asynchronous nature and lack of centralized coordination allows this paradigm to function with limited communication requirements. We demonstrate our method on the MNIST, Fashion MNIST, and CIFAR10 datasets. We show that our coordination method allows models to be learned on highly biased datasets, and in the presence of intermittent communication failure.