Accelerating Training using Tensor Decomposition
This provides a hardware-independent method to accelerate training for machine learning practitioners, though it is incremental as it builds on existing tensor decomposition techniques.
The paper tackles the problem of reducing training time for models trained from scratch by using tensor decomposition, achieving up to 2x speedup with accuracy drops of up to 1.5% or none on CIFAR10 and Imagenet datasets.
Tensor decomposition is one of the well-known approaches to reduce the latency time and number of parameters of a pre-trained model. However, in this paper, we propose an approach to use tensor decomposition to reduce training time of training a model from scratch. In our approach, we train the model from scratch (i.e., randomly initialized weights) with its original architecture for a small number of epochs, then the model is decomposed, and then continue training the decomposed model till the end. There is an optional step in our approach to convert the decomposed architecture back to the original architecture. We present results of using this approach on both CIFAR10 and Imagenet datasets, and show that there can be upto 2x speed up in training time with accuracy drop of upto 1.5% only, and in other cases no accuracy drop. This training acceleration approach is independent of hardware and is expected to have similar speed ups on both CPU and GPU platforms.