Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures
This addresses the problem of computational expense in deep learning for practitioners, though it appears incremental as it builds on existing dimensionality reduction techniques.
The paper tackles the high computational cost of training deep neural networks with large capacity by proposing a novel architecture based on Kronecker product decomposition for dimensionality reduction, achieving similar error levels with significantly reduced computational time and resources.
Deep learning using neural networks is an effective technique for generating models of complex data. However, training such models can be expensive when networks have large model capacity resulting from a large number of layers and nodes. For training in such a computationally prohibitive regime, dimensionality reduction techniques ease the computational burden, and allow implementations of more robust networks. We propose a novel type of such dimensionality reduction via a new deep learning architecture based on fast matrix multiplication of a Kronecker product decomposition; in particular our network construction can be viewed as a Kronecker product-induced sparsification of an "extended" fully connected network. Analysis and practical examples show that this architecture allows a neural network to be trained and implemented with a significant reduction in computational time and resources, while achieving a similar error level compared to a traditional feedforward neural network.