Sparse Activity and Sparse Connectivity in Supervised Learning
This work addresses the challenge of enhancing classification capabilities in neural networks for tasks like digit recognition, representing an incremental improvement through sparsity regularization.
The paper tackles the problem of improving classification performance in neural networks by using sparse activity and sparse connectivity as regularizers, and shows that combining both can significantly boost performance on the MNIST dataset compared to non-sparse methods.
Sparseness is a useful regularizer for learning in a wide range of applications, in particular in neural networks. This paper proposes a model targeted at classification tasks, where sparse activity and sparse connectivity are used to enhance classification capabilities. The tool for achieving this is a sparseness-enforcing projection operator which finds the closest vector with a pre-defined sparseness for any given vector. In the theoretical part of this paper, a comprehensive theory for such a projection is developed. In conclusion, it is shown that the projection is differentiable almost everywhere and can thus be implemented as a smooth neuronal transfer function. The entire model can hence be tuned end-to-end using gradient-based methods. Experiments on the MNIST database of handwritten digits show that classification performance can be boosted by sparse activity or sparse connectivity. With a combination of both, performance can be significantly better compared to classical non-sparse approaches.