Learning Sparse Deep Feedforward Networks via Tree Skeleton Expansion
This addresses the challenge of designing efficient and interpretable neural network architectures for machine learning practitioners, though it is incremental as it builds on existing probabilistic graphical model techniques.
The paper tackles the problem of structure learning for deep feedforward neural networks by proposing a method based on hierarchical latent tree models, resulting in sparse networks that achieve comparable or better classification performance with significantly fewer parameters.
Despite the popularity of deep learning, structure learning for deep models remains a relatively under-explored area. In contrast, structure learning has been studied extensively for probabilistic graphical models (PGMs). In particular, an efficient algorithm has been developed for learning a class of tree-structured PGMs called hierarchical latent tree models (HLTMs), where there is a layer of observed variables at the bottom and multiple layers of latent variables on top. In this paper, we propose a simple method for learning the structures of feedforward neural networks (FNNs) based on HLTMs. The idea is to expand the connections in the tree skeletons from HLTMs and to use the resulting structures for FNNs. An important characteristic of FNN structures learned this way is that they are sparse. We present extensive empirical results to show that, compared with standard FNNs tuned-manually, sparse FNNs learned by our method achieve better or comparable classification performance with much fewer parameters. They are also more interpretable.