An Effective Training Method For Deep Convolutional Neural Network
This addresses training efficiency and stability issues for deep learning practitioners, but it is incremental as it builds on existing activation function modifications.
The paper tackles the problem of slow and unstable training in deep convolutional neural networks by proposing a nonlinearity generation method that modifies activation functions to act as nonlinearity generators, resulting in faster convergence, improved stability, and maintained performance on CIFAR-10 and CIFAR-100 benchmarks at negligible extra cost.
In this paper, we propose the nonlinearity generation method to speed up and stabilize the training of deep convolutional neural networks. The proposed method modifies a family of activation functions as nonlinearity generators (NGs). NGs make the activation functions linear symmetric for their inputs to lower model capacity, and automatically introduce nonlinearity to enhance the capacity of the model during training. The proposed method can be considered an unusual form of regularization: the model parameters are obtained by training a relatively low-capacity model, that is relatively easy to optimize at the beginning, with only a few iterations, and these parameters are reused for the initialization of a higher-capacity model. We derive the upper and lower bounds of variance of the weight variation, and show that the initial symmetric structure of NGs helps stabilize training. We evaluate the proposed method on different frameworks of convolutional neural networks over two object recognition benchmark tasks (CIFAR-10 and CIFAR-100). Experimental results showed that the proposed method allows us to (1) speed up the convergence of training, (2) allow for less careful weight initialization, (3) improve or at least maintain the performance of the model at negligible extra computational cost, and (4) easily train a very deep model.