Deep Learning with Eigenvalue Decay Regularizer
This work provides an incremental improvement to regularization techniques for deep learning practitioners, addressing computational challenges in eigenvalue-based methods.
The authors tackled the problem of applying eigenvalue decay regularization to deep neural networks by developing a soft approximation that enables gradient calculation for backpropagation, achieving competitive performance on standard benchmark datasets including Reuters, IMDB, MNIST, and CIFAR-10.
This paper extends our previous work on regularization of neural networks using Eigenvalue Decay by employing a soft approximation of the dominant eigenvalue in order to enable the calculation of its derivatives in relation to the synaptic weights, and therefore the application of back-propagation, which is a primary demand for deep learning. Moreover, we extend our previous theoretical analysis to deep neural networks and multiclass classification problems. Our method is implemented as an additional regularizer in Keras, a modular neural networks library written in Python, and evaluated in the benchmark data sets Reuters Newswire Topics Classification, IMDB database for binary sentiment classification, MNIST database of handwritten digits and CIFAR-10 data set for image classification.