Neural reproducing kernel Banach spaces and representer theorems for deep networks
This work provides a theoretical foundation for understanding deep networks, addressing a gap in existing results that were limited to shallow networks, which is incremental but important for machine learning theory.
The paper tackles the problem of characterizing the function spaces defined by deep neural networks, showing that they correspond to reproducing kernel Banach spaces with sparsity-enforcing norms, and derives representer theorems to justify finite architectures used in practice.
Characterizing the function spaces defined by neural networks helps understanding the corresponding learning models and their inductive bias. While in some limits neural networks correspond to function spaces that are Hilbert spaces, these regimes do not capture the properties of the networks used in practice. Indeed, several results have shown that shallow networks can be better characterized in terms of suitable Banach spaces. However, analogous results for deep networks are limited. In this paper we show that deep neural networks define suitable reproducing kernel Banach spaces. These spaces are equipped with norms that enforce a form of sparsity, enabling them to adapt to potential latent structures within the input data and their representations. In particular, by leveraging the theory of reproducing kernel Banach spaces, combined with variational results, we derive representer theorems that justify the finite architectures commonly employed in applications. Our study extends analogous results for shallow networks and represents a step towards understanding the function spaces induced by neural architectures used in practice.