Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid
This addresses the bottleneck of training time for deep neural networks, particularly in large-scale or resource-intensive applications, though it appears incremental as it builds on existing multigrid and parallelism concepts.
The paper tackled the problem of slow layer-wise training in deep residual networks by developing a Multigrid Full Approximation Storage algorithm that enables parallelized layer-wise training and concurrent GPU kernel execution, achieving a 10.2x speedup over traditional model parallelism techniques.
A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.