Improving Neural Network Learning Through Dual Variable Learning Rates
This incremental improvement addresses training efficiency for neural network practitioners in domains like image classification.
The paper tackled the problem of neural network training by introducing Dual Variable Learning Rates (DVLR), a method that uses different learning rates for correct and incorrect responses based on performance, resulting in consistently improved accuracy on MNIST and CIFAR-10 datasets.
This paper introduces and evaluates a novel training method for neural networks: Dual Variable Learning Rates (DVLR). Building on insights from behavioral psychology, the dual learning rates are used to emphasize correct and incorrect responses differently, thereby making the feedback to the network more specific. Further, the learning rates are varied as a function of the network's performance, thereby making it more efficient. DVLR was implemented on three types of networks: feedforward, convolutional, and residual, and two domains: MNIST and CIFAR-10. The results suggest a consistently improved accuracy, demonstrating that DVLR is a promising, psychologically motivated technique for training neural network models.