Sparse Neural Networks Topologies
This work addresses the need for more efficient neural network models by proposing pre-defined sparse architectures, offering potential benefits for resource-constrained applications, though it is incremental as it builds on existing sparsity concepts.
The authors tackled the problem of designing sparse neural network architectures by using random or structured bipartite graph topologies, showing that these models can achieve compression and speed-ups while maintaining or surpassing the accuracy of fully connected counterparts, with performance depending on expander-like properties like spectral gap rather than density.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologies of the so-called SNN (Sparse Neural Network) can be achieved with the use of structured graphs of connections between consecutive layers of neurons. In this paper, we investigate how the accuracy and training speed of the models depend on the topology and sparsity of the neural network. Previous approaches using sparcity are all based on fully connected neural network models and create sparcity during training phase, instead we explicitly define a sparse architectures of connections before the training. Building compact neural network models is coherent with empirical observations showing that there is much redundancy in learned neural network models. We show experimentally that the accuracy of the models learned with neural networks depends on expander-like properties of the underlying topologies such as the spectral gap and algebraic connectivity rather than the density of the graphs of connections.