Sparsity-aware generalization theory for deep neural networks
This work addresses the fundamental challenge of explaining generalization in deep learning for researchers, offering a novel theoretical approach that is incremental but improves upon existing methods.
The paper tackles the problem of understanding generalization in deep ReLU networks by developing a sparsity-aware framework that accounts for reduced effective model size per input, showing trade-offs between sparsity and generalization and improving over norm-based approaches, with numerical results demonstrating non-vacuous bounds in over-parametrized models.
Deep artificial neural networks achieve surprising generalization abilities that remain poorly understood. In this paper, we present a new approach to analyzing generalization for deep feed-forward ReLU networks that takes advantage of the degree of sparsity that is achieved in the hidden layer activations. By developing a framework that accounts for this reduced effective model size for each input sample, we are able to show fundamental trade-offs between sparsity and generalization. Importantly, our results make no strong assumptions about the degree of sparsity achieved by the model, and it improves over recent norm-based approaches. We illustrate our results numerically, demonstrating non-vacuous bounds when coupled with data-dependent priors in specific settings, even in over-parametrized models.