Accelerated CNN Training Through Gradient Approximation
This work addresses the computational cost of training deep CNNs for researchers and practitioners, but it is incremental as it builds on prior gradient approximation efforts.
The paper tackled the problem of expensive CNN training by exploring gradient approximation methods, achieving a wall-clock speedup of over 7% on ResNet-20 and VGG-19 with CIFAR-10 while maintaining minimal accuracy loss.
Training deep convolutional neural networks such as VGG and ResNet by gradient descent is an expensive exercise requiring specialized hardware such as GPUs. Recent works have examined the possibility of approximating the gradient computation while maintaining the same convergence properties. While promising, the approximations only work on relatively small datasets such as MNIST. They also fail to achieve real wall-clock speedups due to lack of efficient GPU implementations of the proposed approximation methods. In this work, we explore three alternative methods to approximate gradients, with an efficient GPU kernel implementation for one of them. We achieve wall-clock speedup with ResNet-20 and VGG-19 on the CIFAR-10 dataset upwards of 7%, with a minimal loss in validation accuracy.