Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator
This work provides a theoretical foundation for designing and analyzing deep kernel methods, which could benefit researchers in machine learning, though it appears incremental as it builds on existing concepts like RKHS and kernel methods.
The authors tackled the problem of developing a deep learning framework for kernel methods by combining reproducing kernel Hilbert C*-modules (RKHMs) and Perron-Frobenius operators, resulting in a new Rademacher generalization bound with milder dependency on output dimension and a theoretical interpretation of benign overfitting.
Reproducing kernel Hilbert $C^*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C^*$-algebra, and the Perron-Frobenius operator is a linear operator related to the composition of functions. Combining these two concepts, we present deep RKHM, a deep learning framework for kernel methods. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of Perron-Frobenius operators. By virtue of $C^*$-algebra, the dependency of the bound on output dimension is milder than existing bounds. We show that $C^*$-algebra is a suitable tool for deep learning with kernels, enabling us to take advantage of the product structure of operators and to provide a clear connection with convolutional neural networks. Our theoretical analysis provides a new lens through which one can design and analyze deep kernel methods.