Incremental Training of Deep Convolutional Neural Networks
This work addresses training efficiency for deep learning practitioners, but it is incremental as it builds on existing network architectures and training methods.
The paper tackles the problem of training deep convolutional neural networks more efficiently by proposing an incremental method that partitions networks into sub-networks and gradually incorporates them, achieving baseline accuracy while identifying smaller partitions that use fewer parameters and potentially speed up training by several factors, as demonstrated on CIFAR-10 with ResNet and VGGNet.
We propose an incremental training method that partitions the original network into sub-networks, which are then gradually incorporated in the running network during the training process. To allow for a smooth dynamic growth of the network, we introduce a look-ahead initialization that outperforms the random initialization. We demonstrate that our incremental approach reaches the reference network baseline accuracy. Additionally, it allows to identify smaller partitions of the original state-of-the-art network, that deliver the same final accuracy, by using only a fraction of the global number of parameters. This allows for a potential speedup of the training time of several factors. We report training results on CIFAR-10 for ResNet and VGGNet.