Parareal Neural Networks Emulating a Parallel-in-time Algorithm
This addresses the training time bottleneck for deep learning practitioners, but it appears incremental as it adapts an existing parallel-in-time algorithm to neural networks.
The paper tackles the problem of increasing training time for deep neural networks by introducing a novel methodology to construct parallel neural networks that utilize multiple GPUs simultaneously, emulating a parallel-in-time algorithm called parareal. It reports accelerated and accuracy-preserved results when applied to VGG-16 and ResNet-1001 on several datasets, though specific numbers are not provided.
As deep neural networks (DNNs) become deeper, the training time increases. In this perspective, multi-GPU parallel computing has become a key tool in accelerating the training of DNNs. In this paper, we introduce a novel methodology to construct a parallel neural network that can utilize multiple GPUs simultaneously from a given DNN. We observe that layers of DNN can be interpreted as the time step of a time-dependent problem and can be parallelized by emulating a parallel-in-time algorithm called parareal. The parareal algorithm consists of fine structures which can be implemented in parallel and a coarse structure which gives suitable approximations to the fine structures. By emulating it, the layers of DNN are torn to form a parallel structure which is connected using a suitable coarse network. We report accelerated and accuracy-preserved results of the proposed methodology applied to VGG-16 and ResNet-1001 on several datasets.