Deep Networks are Reproducing Kernel Chains
This work addresses a foundational open question in machine learning by providing a new theoretical framework for deep networks, which is incremental as it extends existing RKBS concepts.
The paper tackles the problem of identifying a function space for deep neural networks by introducing chain Reproducing Kernel Banach Spaces (cRKBS), which composes kernels instead of functions. It proves that deep neural network functions are neural cRKBS functions and provides a sparse solution to empirical risk minimization with at most N neurons per layer, where N is the number of data points.
Identifying an appropriate function space for deep neural networks remains a key open question. While shallow neural networks are naturally associated with Reproducing Kernel Banach Spaces (RKBS), deep networks present unique challenges. In this work, we extend RKBS to chain RKBS (cRKBS), a new framework that composes kernels rather than functions, preserving the desirable properties of RKBS. We prove that any deep neural network function is a neural cRKBS function, and conversely, any neural cRKBS function defined on a finite dataset corresponds to a deep neural network. This approach provides a sparse solution to the empirical risk minimization problem, requiring no more than $N$ neurons per layer, where $N$ is the number of data points.