Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
This addresses the challenge of exploiting parallel computing trends for machine learning practitioners, offering a potentially transformative approach rather than an incremental improvement.
The paper tackles the problem of inefficient parallelization in deep neural network training due to sequential gradient descent, and demonstrates that replicating DNNs over a parallel architecture with convolutional bootstrapping enables rapid learning, outperforming a fully-trained traditional DNN after just one iteration.
Although deep neural networks (DNN) are able to scale with direct advances in computational power (e.g., memory and processing speed), they are not well suited to exploit the recent trends for parallel architectures. In particular, gradient descent is a sequential process and the resulting serial dependencies mean that DNN training cannot be parallelized effectively. Here, we show that a DNN may be replicated over a massive parallel architecture and used to provide a cumulative sampling of local solution space which results in rapid and robust learning. We introduce a complimentary convolutional bootstrapping approach that enhances performance of the parallel architecture further. Our parallelized convolutional bootstrapping DNN out-performs an identical fully-trained traditional DNN after only a single iteration of training.