Statistical guarantees for sparse deep learning
This work provides theoretical foundations for sparse deep learning, benefiting researchers and practitioners by supporting the use of wide and deep networks statistically, though it is incremental in extending existing theories.
The paper tackles the problem of limited mathematical understanding of neural networks by developing statistical guarantees for sparse deep learning, covering various sparsity types and aspects like multiple outputs and regularization, with mild dependence on network dimensions.
Neural networks are becoming increasingly popular in applications, but our mathematical understanding of their potential and limitations is still limited. In this paper, we further this understanding by developing statistical guarantees for sparse deep learning. In contrast to previous work, we consider different types of sparsity, such as few active connections, few active nodes, and other norm-based types of sparsity. Moreover, our theories cover important aspects that previous theories have neglected, such as multiple outputs, regularization, and l2-loss. The guarantees have a mild dependence on network widths and depths, which means that they support the application of sparse but wide and deep networks from a statistical perspective. Some of the concepts and tools that we use in our derivations are uncommon in deep learning and, hence, might be of additional interest.