Building Sparse Deep Feedforward Networks using Tree Receptive Fields
This addresses the need for more efficient and interpretable neural networks, though it is incremental as it adapts existing methods to a specific architecture.
The paper tackles the problem of learning sparse connectivity for feedforward neural networks by using Chow-Liu's algorithm to build tree-structured models, resulting in TRF-nets that achieve better or comparable classification performance with much fewer parameters and sparser structures compared to dense FNNs.
Sparse connectivity is an important factor behind the success of convolutional neural networks and recurrent neural networks. In this paper, we consider the problem of learning sparse connectivity for feedforward neural networks (FNNs). The key idea is that a unit should be connected to a small number of units at the next level below that are strongly correlated. We use Chow-Liu's algorithm to learn a tree-structured probabilistic model for the units at the current level, use the tree to identify subsets of units that are strongly correlated, and introduce a new unit with receptive field over the subsets. The procedure is repeated on the new units to build multiple layers of hidden units. The resulting model is called a TRF-net. Empirical results show that, when compared to dense FNNs, TRF-net achieves better or comparable classification performance with much fewer parameters and sparser structures. They are also more interpretable.