Layer Sparsity in Neural Networks
This work addresses the need for efficient and interpretable deep learning models, but it appears incremental as it builds on existing sparsity concepts by applying them specifically to layers.
The paper tackles the problem of making neural networks more compact and accurate by introducing a new notion of sparsity focused on layers, which is particularly relevant for deep networks, and proposes regularization and refitting schemes to achieve this.
Sparsity has become popular in machine learning, because it can save computational resources, facilitate interpretations, and prevent overfitting. In this paper, we discuss sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity that concerns the networks' layers and, therefore, aligns particularly well with the current trend toward deep networks. We call this notion layer sparsity. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.