Hypothesis Spaces for Deep Learning
This provides a theoretical foundation for deep learning models, addressing a core mathematical problem for researchers in machine learning theory, though it is incremental in extending existing kernel methods to deep networks.
The paper tackles the problem of formalizing a hypothesis space for deep learning by constructing a Banach space from deep neural networks, proving it is a reproducing kernel Banach space and deriving explicit kernels. It shows that solutions to regularized learning and minimum norm interpolation problems can be expressed as finite kernel expansions based on training data.
This paper introduces a hypothesis space for deep learning based on deep neural networks (DNNs). By treating a DNN as a function of two variables - the input variable and the parameter variable - we consider the set of DNNs where the parameter variable belongs to a space of weight matrices and biases determined by a prescribed depth and layer widths. To construct a Banach space of functions of the input variable, we take the weak* closure of the linear span of this DNN set. We prove that the resulting Banach space is a reproducing kernel Banach space (RKBS) and explicitly construct its reproducing kernel. Furthermore, we investigate two learning models - regularized learning and the minimum norm interpolation (MNI) problem - within the RKBS framework by establishing representer theorems. These theorems reveal that the solutions to these learning problems can be expressed as a finite sum of kernel expansions based on training data.