Analysis of Structured Deep Kernel Networks
This work provides a theoretical framework for analyzing neural networks through kernel methods, which is incremental in bridging two established areas of machine learning.
The paper connects kernel methods and deep neural networks by introducing Structured Deep Kernel Networks (SDKNs), which are shown to have universal approximation properties and can achieve more accurate constructions with fewer layers than ReLU networks in the unbounded depth regime.
In this paper, we leverage a recent deep kernel representer theorem to connect kernel based learning and (deep) neural networks in order to understand their interplay. In particular, we show that the use of special types of kernels yields models reminiscent of neural networks that are founded in the same theoretical framework of classical kernel methods, while benefiting from the computational advantages of deep neural networks. Especially the introduced Structured Deep Kernel Networks (SDKNs) can be viewed as neural networks (NNs) with optimizable activation functions obeying a representer theorem. This link allows us to analyze also NNs within the framework of kernel networks. We prove analytic properties of the SDKNs which show their universal approximation properties in three different asymptotic regimes of unbounded number of centers, width and depth. Especially in the case of unbounded depth, more accurate constructions can be achieved using fewer layers compared to corresponding constructions for ReLU neural networks. This is made possible by leveraging properties of kernel approximation.