Tensor Network-Constrained Kernel Machines as Gaussian Processes
This work provides a theoretical link between tensor networks and Gaussian processes, which is incremental but clarifies the behavior of constrained kernel machines for machine learning researchers.
The paper tackles the problem of connecting tensor network-constrained kernel machines to Gaussian processes, proving that models using Canonical Polyadic Decomposition and Tensor Train constraints recover a Gaussian process under i.i.d. priors, with Tensor Train showing more Gaussian process behavior for the same parameter count.
Tensor Networks (TNs) have recently been used to speed up kernel machines by constraining the model weights, yielding exponential computational and storage savings. In this paper we prove that the outputs of Canonical Polyadic Decomposition (CPD) and Tensor Train (TT)-constrained kernel machines recover a Gaussian Process (GP), which we fully characterize, when placing i.i.d. priors over their parameters. We analyze the convergence of both CPD and TT-constrained models, and show how TT yields models exhibiting more GP behavior compared to CPD, for the same number of model parameters. We empirically observe this behavior in two numerical experiments where we respectively analyze the convergence to the GP and the performance at prediction. We thereby establish a connection between TN-constrained kernel machines and GPs.